메뉴 건너뛰기

Bigdata, Semantic IoT, Hadoop, NoSQL

Bigdata, Hadoop ecosystem, Semantic IoT등의 프로젝트를 진행중에 습득한 내용을 정리하는 곳입니다.
필요한 분을 위해서 공개하고 있습니다. 문의사항은 gooper@gooper.com로 메일을 보내주세요.


기타 ubuntu에 hadoop 2.0.5설치하기

총관리자 2013.12.16 22:09 조회 수 : 1879

출처 : http://www.spikyjohn.com/cribsheets/20130609_hadoopinstall.html

 

Just the command lines to get hadoop 2 installed on Ubuntu. These are all cribbed from the following source notes, and I am preserving them here for my own benefit so I can quickly repeat what I did. Note many of these instructions are also in the main hadoop docs from apache.

Source material

Use Michael-noll's guide for version 1 & ssh
http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster/
http://hadoop.apache.org/docs/r1.1.2/single_node_setup.html

Or this one for Hadoop 2
http://jugnu-life.blogspot.com/2012/05/hadoop-20-install-tutorial-023x.html
http://hadoop.apache.org/docs/r2.0.5-alpha/

Create the hadoop user and ssh

sudo apt-get install openssh-server openssh-client

sudo addgroup hadoop
sudo adduser --ingroup hadoop hduser
su - hduser

If you cannot ssh to localhost without a passphrase, execute the following commands:
ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

Testing your SSH
ssh localhost
Say yes
#exit

Get hadoop all set up

As the hduser, after downloading the tar

tar -xvf hadoop-2.0.5-alpha.tar.gz
ln -s hadoop-2.0.5-alpha hadoop
#edit .bashrc
export JAVA_HOME=/usr/lib/jvm/jdk1.7.0_21/
export HADOOP_PREFIX="/home/hduser/hadoop"
export PATH=$PATH:$HADOOP_PREFIX/bin
export PATH=$PATH:$HADOOP_PREFIX/sbin

export HADOOP_MAPRED_HOME=${HADOOP_PREFIX}
export HADOOP_COMMON_HOME=${HADOOP_PREFIX}
export HADOOP_HDFS_HOME=${HADOOP_PREFIX}
export YARN_HOME=${HADOOP_PREFIX}

Stolen entirely from JJ, but with path changed for my Ubuntu

Stolen from http://jugnu-life.blogspot.com/2012/05/hadoop-20-install-tutorial-023x.html Please click on his blog.

Login again so bash has paths above. In Hadoop 2.x version /etc/hadoop is the default conf directory. We need to modify / create following property files in the /etc/hadoop directory

cd ~
mkdir -p /home/hduser/workspace/hadoop_space/hadoop23/dfs/name;mkdir -p /home/hduser/workspace/hadoop_space/hadoop23/dfs/data;mkdir -p /home/hduser/workspace/hadoop_space/hadoop23/mapred/system;mkdir -p /home/hduser/workspace/hadoop_space/hadoop23/mapred/local

Edit core-site.xml with following contents

<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:8020</value>
<description>The name of the default file system. Either the literal string "local" or a host:port for NDFS.</description>
<final>true</final>
</property>
</configuration>

Edit hdfs-site.xml with following contents

<configuration>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/home/hduser/workspace/hadoop_space/hadoop23/dfs/name</value>
<description>Determines where on the local filesystem the DFS name node
should store the name table. If this is a comma-delimited list of directories then the name table is replicated in all of the
directories, for redundancy. </description>
<final>true</final>
</property>

<property>
<name>dfs.datanode.data.dir</name>
<value>file:/home/hduser/workspace/hadoop_space/hadoop23/dfs/data</value>
<description>Determines where on the local filesystem an DFS data node
should store its blocks. If this is a comma-delimited list of directories, then data will be stored in all named
directories, typically on different devices. Directories that do not exist are ignored.
</description>
<final>true</final>
</property>

<property>
<name>dfs.replication</name>
<value>1</value>
</property>

<property>
<name>dfs.permissions</name>
<value>false</value>
</property>

</configuration>

The path
file:/home/hduser/workspace/hadoop_space/hadoop23/dfs/name AND
file:/home/hduser/workspace/hadoop_space/hadoop23/dfs/data
are some folders in your computer which would give space to store data and name edit files

Path should be specified as URI
Create a file mapred-site.xml inside /etc/hadoop with following contents

<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>

<property>
<name>mapred.system.dir</name>
<value>file:/home/hduser/workspace/hadoop_space/hadoop23/mapred/system</value>
<final>true</final>
</property>

<property>
<name>mapred.local.dir</name>
<value>file:/home/hduser/workspace/hadoop_space/hadoop23/mapred/local</value>
<final>true</final>
</property>

</configuration>

The path

file:/home/hduser/workspace/hadoop_space/hadoop23/mapred/system AND
file:/home/hduser/workspace/hadoop_space/hadoop23/mapred/local
are some folders in your computer which would give space to store data

Path should be specified as URI

Edit yarn-site.xml with following contents

<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce.shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>

Format the namenode

# hdfs namenode –format

Say Yes and let it complete the format

Time to start the daemons

# hadoop-daemon.sh start namenode
# hadoop-daemon.sh start datanode

You can also start both of them together by

# start-dfs.sh

Start Yarn Daemons

# yarn-daemon.sh start resourcemanager
# yarn-daemon.sh start nodemanager

You can also start all yarn daemons together by

# start-yarn.sh

Time to check if Daemons have started

Enter the command

# jps
2539 NameNode
2744 NodeManager
3075 Jps
3030 DataNode
2691 ResourceManager

Time to launch UI

Open the localhost:8088 to see the Resource Manager page

Done :)

Happy Hadooping :)

번호 제목 글쓴이 날짜 조회 수
680 hadoop설치시 참고사항 구퍼 2013.03.08 2131
679 solr설치및 적용관련 file 총관리자 2014.09.27 2124
678 W/F수행후 Logs not available for 1. Aggregation may not to complete. 표시되며 로그내용이 보이지 않은 경우 총관리자 2020.05.08 2110
677 hbase에 필요한 jar들 구퍼 2013.04.01 2100
676 ../depcomp: line 512 exec : g++ : not found 구퍼 2013.03.08 2062
675 Hive java connection 설정 file 구퍼 2013.04.01 2013
674 RHadoop을 통해서 웹사이트의 방문자수를 예측하는 방법 총관리자 2014.09.26 2007
673 통계자료 구할수 있는 곳 총관리자 2014.04.16 1964
672 Hadoop 설치 및 시작하기 file 구퍼 2013.03.06 1951
671 hbase shell 필드 검색 방법 총관리자 2015.05.24 1900
670 VisualVM 1.3.9을 이용한 spark-submit JVM 모니터링을 위한 설정및 spark-submit실행 옵션 총관리자 2016.10.28 1891
669 Hadoop wordcount 소스 작성 file 구퍼 2013.03.06 1888
668 Spark 2.1.1 clustering(5대) 설치(YARN기반) 총관리자 2016.04.22 1882
» ubuntu에 hadoop 2.0.5설치하기 총관리자 2013.12.16 1879
666 MySQL 다운로드 및 리눅스에서 간단 컴파일 설치 구퍼 2013.03.08 1869
665 hadoop 2.6.0에 sqoop2 (1.99.5) server및 client설치 == fail 총관리자 2015.06.11 1770
664 VirtualBox에 ubuntu 설치 하기 (12.10) file 구퍼 2013.03.04 1768
663 hive에서 생성된 external table에서 hbase의 table에 값 insert하기 총관리자 2014.04.11 1748
662 [sbt] sbt-assembly를 이용하여 실행에 필요한 모든 j라이브러리를 포함한 fat jar파일 만들기 총관리자 2016.07.11 1738
661 access=WRITE, inode="staging":ubuntu:supergroup:rwxr-xr-x 오류 총관리자 2014.07.05 1719

A personal place to organize information learned during the development of such Hadoop, Hive, Hbase, Semantic IoT, etc.
We are open to the required minutes. Please send inquiries to gooper@gooper.com.

위로