详细版在虚拟机安装和使用hadoop分布式集群

  集群模式:

  一台master   192.168.85.2

  一台slave    192.168.85.3 

  jdk      jdk1.8.0_74(版本不重要,看喜欢)

  hadoop版本  2.7.2(版本不重要,2.*都差不多)

  本文从安装Ubuntu14.04后开始一步步搭建hadoop集群:

  简单说一下虚拟机linux系统的搭建:

  因为要搭建hadoop集群,所以预计至少两台虚拟机,这个不着急,我们可以布置一台然后克隆出另一台,然后稍微改动一下配置

  我用的镜像是ubuntu-14.04.3-server-amd64.iso,为了主机连接和网络连接建立两个网卡,相关内容可以查看另外一篇博文:本机上搭建虚拟机的网络玩法,安装过程中注意安装openssh服务就好了,安装好之后可以用工具ssh到虚拟机上面操作更方便.安装的时候可以直接指定主机名为master比较好识别,用户名指定为hadoop

  1.安装jdk

  查看是否安装jdk

java -version

  如果未安装参考:Ubuntu系统如何卸载并安装新版本的jdk(permission denied问题),已安装则跳过此步

  2.下载hadoop

  我下载的地址http://mirror.bit.edu.cn/apache/hadoop/common/,上面会有很多种版本可以选择,对试用来说都是一样的.随便下一个

  通过ftp或者ssh传送到虚拟机上解压:

tar zxvf hadoop-2.7.2.tar.gz

  重命名:

mv hadoop-2.7.2 hadoop

  查看安装目录:

hadoop@master:~/hadoop$ pwd
/home/hadoop/hadoop

  接下来配置多个配置文件,配置文件集中在安装目录下的的etc/hadoop下,我们将目录切换到该目录下方便操作,我将配置的内容贴出来:

  slaves文件

vi slaves

  内容改为

master

  core-site.xml文件

vi core-site.xml

  在<configuration>标签中添加如下内容:

<property>
        <name>hadoop.tmp.dir</name>
        <value>/home/hadoop/hadoop/tmp</value>
        <description>Abase for other temporary directories.</description>
    </property>
    <property>
        <name>fs.default.name</name>
        <value>hdfs://master:9000</value>
    </property>

  hdfs-site.xml

vi hdfs-site.xml

  添加:

<property>
    <name>dfs.name.dir</name>
    <value>/home/hadoop/hadoop/dfs/name</value>
    <description>Path on the local filesystem where the NameNode stores the namespace and transactions logs persistently.</description>
</property>

<property>
    <name>dfs.data.dir</name>
    <value>/home/hadoop/hadoop/dfs/data</value>
    <description>Comma separated list of paths on the local filesystem of a DataNode where it should store its blocks.</description>
</property>
<property>
    <name>dfs.replication</name>
    <value>1</value>
</property>

  mapred-site.xml,这个文件需要从模板中复制一份过来:

cp mapred-site.xml.template mapred-site.xml 
vi mapred-site.xml

  添加

<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
</property>

  yarn-site.xml

vi yarn-site.xml

  添加:

<property>
    <name>yarn.resourcemanager.hostname</name>
    <value>master</value>
</property>
<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>

  到这边理论上hadoop已经可以跑了.但是在hadoop启动过程中因为脚本的限定可能会报一些环境配置错误,我经过实践为了一绝后患先将该配置的东西配置好

  首先是的java_home配置:

vi hadoop-env.sh

  修改

export JAVA_HOME=/home/hadoop/jdk1.8.0_74

  然后添加hadoop环境变量配置(在java配置下面添加就行):

export HADOOP_DEV_HOME=/home/hadoop/hadoop
export PATH=$PATH:$HADOOP_DEV_HOME/bin
export PATH=$PATH:$HADOOP_DEV_HOME/sbin
export HADOOP_MAPARED_HOME=${HADOOP_DEV_HOME}
export HADOOP_COMMON_HOME=${HADOOP_DEV_HOME}
export HADOOP_HDFS_HOME=${HADOOP_DEV_HOME}
export YARN_HOME=${HADOOP_DEV_HOME}
export HADOOP_CONF_DIR=${HADOOP_DEV_HOME}/etc/hadoop
export HDFS_CONF_DIR=${HADOOP_DEV_HOME}/etc/hadoop
export YARN_CONF_DIR=${HADOOP_DEV_HOME}/etc/hadoop

  保存更改

source ~/.bashrc

  格式化hdfs

bin/hdfs namenode -format

  在克隆虚拟机之前先将主机配置好.

vi /etc/hosts

  修改

127.0.0.1       localhost
#127.0.1.1      master
192.168.85.2    master
192.168.85.3    slave1

  克隆虚拟机,并启动克隆的机器.修改主机名和ip

vi /etc/hostname

  修改为slave1

vi /etc/network/interfaces

  看到

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet dhcp
auto eth1
iface eth1 inet static
address 192.168.85.2
netmask 255.255.255.0

  修改为

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet dhcp
auto eth1
iface eth1 inet static
address 192.168.85.3
netmask 255.255.255.0

  重启机器

  到此,hadoop集群就搭建完了.

  安装两台机器后,需要让master无密码登录到slave上面

ssh localhost
cd ~/.ssh 
ssh-keygen -t rsa 

  一直确认即可;

  Master 节点需能无密码 ssh 本机,这一步还是在 Master 节点上执行:

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

  完成后可以使用 ssh Master 验证一下。接着将公匙传输到 Slave1 节点:

scp ~/.ssh/id_rsa.pub hadoop@Slave1:/home/hadoop/

  scp时会要求输入Slave1上hadoop用户的密码(hadoop),输入完成后会提示传输完毕。

  接着在 Slave1节点 上将ssh公匙保存到相应位置,执行

cat ~/id_rsa.pub >> ~/.ssh/authorized_keys

 

  我们来测试一下是否可以运行:

sbin/start-dfs.sh
sbin/start-yarn.sh

  这个命令启动了master和slave上面的东西,用jps查看内容

hadoop@master:~$ jps
1632 SecondaryNameNode
4581 Jps
1782 ResourceManager
1402 NameNode

  在slave1中执行jps

4586 Jps
3210 DataNode
3356 NodeManager 

  登录http://192.168.85.2:50070/可以看到master和slave的分布以及启动状况

   执行经典案例wordcount.

  新建一个text1.txt并上传到集群

cd
mkdir input
cd input
echo "hello world" > test1.txt
hadoop fs –mkdir input 

  最后一条命令可能会报错,报错找不到input文件夹,那是因为hdfs初始化还没有根目录,加上/就好了

hadoop fs –mkdir /input

  查看文件:

hadoop@master:~/hadoop$ hadoop fs -ls /
Found 1 items
drwxr-xr-x   - hadoop supergroup          0 2016-05-10 10:36 /input

  上传文件到input中并查看

hadoop@master:~/hadoop$ hadoop fs -put ../input/*.txt /input
hadoop@master:~/hadoop$ hadoop fs -ls /
Found 1 items
drwxr-xr-x - hadoop supergroup 0 2016-05-10 10:38 /input
hadoop@master:~/hadoop$ hadoop fs -ls /input
Found 2 items
-rw-r--r-- 1 hadoop supergroup 12 2016-05-10 10:38 /input/test1.txt
-rw-r--r-- 1 hadoop supergroup 13 2016-05-10 10:38 /input/test2.txt

  接下来就是用hadoop自带的一个脚本运行该文件,计算单词数

hadoop/bin/hadoop jar hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar wordcount /input/test1.txt output2

  格式是hadoop脚本+jar命令+jar脚本+方法+输入文件+输出文件.

  job开始执行输出

16/05/10 10:44:14 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.85.2:8032
16/05/10 10:44:15 INFO input.FileInputFormat: Total input paths to process : 1
16/05/10 10:44:15 INFO mapreduce.JobSubmitter: number of splits:1
16/05/10 10:44:15 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1462879083278_0001
16/05/10 10:44:16 INFO impl.YarnClientImpl: Submitted application application_1462879083278_0001
16/05/10 10:44:16 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1462879083278_0001/
16/05/10 10:44:16 INFO mapreduce.Job: Running job: job_1462879083278_0001
16/05/10 10:44:30 INFO mapreduce.Job: Job job_1462879083278_0001 running in uber mode : false
16/05/10 10:44:30 INFO mapreduce.Job:  map 0% reduce 0%
16/05/10 10:44:40 INFO mapreduce.Job:  map 100% reduce 0%
16/05/10 10:44:47 INFO mapreduce.Job:  map 100% reduce 100%
16/05/10 10:44:47 INFO mapreduce.Job: Job job_1462879083278_0001 completed successfully
16/05/10 10:44:47 INFO mapreduce.Job: Counters: 49
    File System Counters
        FILE: Number of bytes read=30
        FILE: Number of bytes written=234875
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=111
        HDFS: Number of bytes written=16
        HDFS: Number of read operations=6
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=2
    Job Counters 
        Launched map tasks=1
        Launched reduce tasks=1
        Data-local map tasks=1
        Total time spent by all maps in occupied slots (ms)=7470
        Total time spent by all reduces in occupied slots (ms)=4602
        Total time spent by all map tasks (ms)=7470
        Total time spent by all reduce tasks (ms)=4602
        Total vcore-milliseconds taken by all map tasks=7470
        Total vcore-milliseconds taken by all reduce tasks=4602
        Total megabyte-milliseconds taken by all map tasks=7649280
        Total megabyte-milliseconds taken by all reduce tasks=4712448
    Map-Reduce Framework
        Map input records=1
        Map output records=2
        Map output bytes=20
        Map output materialized bytes=30
        Input split bytes=99
        Combine input records=2
        Combine output records=2
        Reduce input groups=2
        Reduce shuffle bytes=30
        Reduce input records=2
        Reduce output records=2
        Spilled Records=4
        Shuffled Maps =1
        Failed Shuffles=0
        Merged Map outputs=1
        GC time elapsed (ms)=229
        CPU time spent (ms)=2730
        Physical memory (bytes) snapshot=298352640
        Virtual memory (bytes) snapshot=3748110336
        Total committed heap usage (bytes)=139145216
    Shuffle Errors
        BAD_ID=0
        CONNECTION=0
        IO_ERROR=0
        WRONG_LENGTH=0
        WRONG_MAP=0
        WRONG_REDUCE=0
    File Input Format Counters 
        Bytes Read=12
    File Output Format Counters 
        Bytes Written=16

  http://master:8088/proxy/application_1462879083278_0001/可以查看当前job的运行状态,在运行过程中可以查看.看到map 100% reduce 100%就是运行成功了,可以登录http://192.168.85.2:8088/cluster查看具体信息

  最后关闭hadoop集群

sbin/stop-dfs.sh
sbin/stop-yarn.sh

  是不是很简单呢.

  

  

  

  

 

 

 

 

 

  

posted @ 2016-05-10 17:34  但行好事-莫问前程  阅读(1007)  评论(0编辑  收藏  举报