hadoop2.5.2安装

一、core-site.xml:包含hadoop启动时的信息。

<configuration>

<property>

<name>fs.default.name</name>

<value>hdfs://localhost:9000</value>

</property>

<property>

<name>hadoop.tmp.dir</name>

<value>/home/xxlong/myInstall/hadoop/tmp</value>

</property>

</configuration>

二、hdfs-site.xml:指定主机namenode和datanode的目录

<configuration>

<property>

<name>dfs.namenode.name.dir</name>

<value>file:/home/xxlong/myInstall/hadoop/hdfs/name</value>

</property>

<property>

<name>dfs.namenode.data.dir</name>

<value>file:/home/xxlong/myInstall/hadoop/hdfs/data</value>

</property>

<property>

<name>dfs.replication</name>

<value>1</value>

</property>

</configuration>

三、mapred-site.xml:复制mapred.xml.template文件命名为mapred-site.xml，指定 MapReduce使用的框架。

<configuration>

<property>

<name>mapreduce.framework.name</name>

<value>yarn</value>

</property>

</configuration>

四、yarn-site.xml:包含了MapReduce启动时的配置信息。

<configuration>

<property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

</configuration>

五、hadoop-env.sh

加入export JAVA_HOME=/home/xxlong/myInstall/jdk1.8.0_45

export HADOOP_LOG_DIR=/home/xxlong/myInstall/hadoop/logs

.profile

export HADOOP_HOME=/home/xxlong/myInstall/hadoop/hadoop-2.5.2

PATH:$HADOOP-HOME/bin

六、ssh免密码登陆

1、sudo apt-get install openssh-server

2、运行 ps -e | grep ssh，查看是否有sshd进程，如果没有/etc/init.d/ssh -start

3、ssh-keygen -t rsa，一路回车

4、cat id_rsa.pub>>authorized_keys

5、chmod 600 authorized_keys

6、如果是远程两个主机A和B

scp /home/A/.ssh/id_rsa.pub B@192.168.1.181:/home/B/.ssh/id_rsa.pub

七、格式化hdfs

hdfs namenode -format

八、启动hdfs

start-dfs.sh

或hadoop-daemon.sh start namenode

hadoop-daemon.sh start datanode

hadoop-daemon.sh start secondarynamenode

九、启动yarn

start-yarn.sh

或hadoop-daemon.sh start resourcemanager

hadoop-daemon.sh start nodemanager

十、hdfs管理界面：http://localhost:50070/

yarn管理界面：http://localhost:8088/cluster

十一、关闭hdfs

stop-dfs.sh

或hadoop-daemon.sh stop namenode

hadoop-daemon.sh stop datanode

hadoop-daemon.sh stop secondarynamenode

十二、stop-yarn.sh

或yarn-daemon.sh stop resourcemanager

yarn-daemon.sh stop nodemanager

十三、eclipse中安装hadoop插件

将hadoop.jar插件复制到eclipse安装目录下的plugins目录中。

1、Windows-->Preferences-->HadoopMap/Reduce-->Hadoop installation

direction:/home/xxlong/myInstall/hadoop/hadoop-2.5.2

2、windows-->showView-->Map/ReduceLocation

3、视图中右击-->New Hadoop location-->Map/Reduce(V2)Master Host:localhost Port:9001 Use M/R Master host Host:localhost Port:9000

 

4、先创建目录，不然报错。

hadoop fs -mkdir -p /user/xxlong/input

hadoop fs -mkdir -p /user/xxlong/output

十四、查看hadoop码源

新建一个java项目,将/hadoop-2.5.2/share/hadoop/mapreduce/sources下hadoop-mapreduce- examples-2.5.2-sources.jar解压后org包复制到新建项目src包下。

将hadoop-2.5.2-src包压缩成zip格式。eclipse中通过点击类按F3引入

十五、运行wordcount

Arguments:

hdfs://localhost:9000/user/xxlong/input/xxlong.txt hdfs://localhost:9000/user/xxlong/output

VM arguments:

Djava.library.path=/home/xxlong/myInstall/hadoop/hadoop-2.5.2/lib/native