Hadoop单节点集群

Hadoop单节点集群

官方网站:


环境:
CentOS 7.1
hadoop-2.7.2
jdk-1.8

说明:仅个人测试学习记录，大神们可以一笑而过

一.jdk
jdk版本，hadoop官方推荐HadoopJavaVersions


二.安装hadoop
1.创建用户
useradd -r -m hadoop
echo hadoop:hd.123|chpasswd
说明:服务启停都由hadoop用户来操作
2.调整资源限制
cat >/etc/security/limits.d/99-hadoop-nproc.conf <<HERE
hadoop soft fsize unlimited 
hadoop soft cpu unlimited 
hadoop soft as unlimited 
hadoop soft nofile 64000 
hadoop soft nproc 64000 
hadoop hard fsize unlimited 
hadoop hard cpu unlimited 
hadoop hard as unlimited 
hadoop hard nofile 64000 
hadoop hard nproc 64000 
HERE
3.安装
tar -xvf hadoop-2.7.2.tar.gz -C /opt
mv /opt/hadoop-2.7.2 /opt/hadoop
chown -R hadoop: /opt/hadoop


三.环境变量
cat >/etc/profile.d/hadoop.sh <<HERE

export JAVA_HOME=/opt/jdk


export HADOOP_PREFIX=/opt/hadoop

export HADOOP_HOME=/opt/hadoop

export HADOOP_CONF_DIR=/opt/hadoop/etc/hadoop

export HADOOP_NAMENODE_OPTS="-XX:+UseParallelGC"

export HADOOP_PORTMAP_OPTS="-Xmx1024m"

export HADOOP_CLIENT_OPTS="-Xmx1024m"

export HADOOP_PID_DIR=/opt/hadoop/run

export HADOOP_SECURE_DN_PID_DIR=/opt/hadoop/run

export HADOOP_LOG_DIR=/opt/hadoop/logs


export HADOOP_YARN_HOME=/opt/hadoop

export YARN_PID_DIR=/opt/hadoop/run


export HADOOP_MAPRED_PID_DIR=/opt/hadoop/run

HERE

source /etc/profile



四.配置HDFS 
1.修改配置文件
etc/hadoop/core-site.xml
Hadoop单节点集群

etc/hadoop/hdfs-site.xml

Hadoop单节点集群

2.ssh密钥认证

su - hadoop

ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa

cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

chmod 0600 ~/.ssh/authorized_keys

 

说明:$HADOOP_PREFIX/sbin/start-dfs.sh脚本会通过ssh连接调用，所以要密钥认证





3.格式化hdfs





$HADOOP_PREFIX/bin/hdfs namenode -format










4.namenode & datanode





$HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start namenode



$HADOOP_PREFIX/sbin/hadoop-daemons.sh --config $HADOOP_CONF_DIR --script hdfs start datanode




或者




当配置了ssh密码认证后可以直接通过脚本启动和停止




启动




$HADOOP_PREFIX/sbin/start-dfs.sh






[hadoop@node-a ~]$
$HADOOP_PREFIX/sbin/start-dfs.sh


Starting
namenodes on [localhost]


localhost:
starting namenode, logging to
/opt/hadoop/logs/hadoop-hadoop-namenode-node-a.example.com.out


localhost:
starting datanode, logging to
/opt/hadoop/logs/hadoop-hadoop-datanode-node-a.example.com.out


Starting
secondary namenodes [0.0.0.0]


0.0.0.0:
starting secondarynamenode, logging to
/opt/hadoop/logs/hadoop-hadoop-secondarynamenode-node-a.example.com.out


注意:主机名要能解析，否则启动报错







停止




$HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs stop namenode



$HADOOP_PREFIX/sbin/hadoop-daemons.sh --config $HADOOP_CONF_DIR --script hdfs stop datanode






或者




当配置了ssh密码认证后可以直接通过脚本启动和停止




$HADOOP_PREFIX/sbin/stop-dfs.sh







[hadoop@node-a
~]$ HADOOP_PREFIX/sbin/stop-dfs.sh



Stopping
namenodes on [localhost]



localhost: stopping
namenode



localhost: stopping
datanode



Stopping
secondary namenodes [0.0.0.0]



0.0.0.0:
stopping secondarynamenode






5.NameNode web接口


localhost:50070



Hadoop单节点集群












四.配置YARN


1.修改配置文件


etc/hadoop/mapred-site.xml


Hadoop单节点集群






etc/hadoop/yarn-site.xml


Hadoop单节点集群





2.ssh密钥认证







同上，略











3.resourcemanager & nodemanager & historyserver










启动




$HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start resourcemanager







$HADOOP_YARN_HOME/sbin/yarn-daemons.sh --config $HADOOP_CONF_DIR start nodemanager



$HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start proxyserver








$HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh --config $HADOOP_CONF_DIR start historyserver










或者




当配置了ssh密码认证后可以直接通过脚本启动和停止



$HADOOP_PREFIX/sbin/start-yarn.sh






提示:start-yarn.sh最后一行的start proxyserver默认是注释状态，可以按需启用



$HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh --config $HADOOP_CONF_DIR start historyserver











[hadoop@node-a
~]$ $HADOOP_PREFIX/sbin/start-yarn.sh



starting
yarn daemons



starting
resourcemanager, logging to
/opt/hadoop/logs/yarn-root-resourcemanager-ct7.example.com.out



localhost: starting
nodemanager, logging to
/opt/hadoop/logs/yarn-root-nodemanager-ct7.example.com.out



 







停止









$HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR stop resourcemanager







$HADOOP_YARN_HOME/sbin/yarn-daemons.sh --config $HADOOP_CONF_DIR stop nodemanager



$HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR stop proxyserver





$HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh --config $HADOOP_CONF_DIR stop historyserver








或者




当配置了ssh密码认证后可以直接通过脚本启动和停止




$HADOOP_PREFIX/sbin/stop-yarn.sh






$HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh --config $HADOOP_CONF_DIR stop historyserver












[hadoop@node-a
~]$ HADOOP_PREFIX/sbin/stop-yarn.sh



stopping
yarn daemons



stopping
resourcemanager



localhost: stopping
nodemanager



no
proxyserver to stop


[hadoop@node-a
~]$ netstat -tunlp|grep
java


tcp  
     0
     0
0.0.0.0:8033    
     
 0.0.0.0:*    
     
    LISTEN  
   17045/java


tcp  
     0
     0
0.0.0.0:10020    
      0.0.0.0:*
     
     
  LISTEN    
 17493/java


tcp  
     0
     0
0.0.0.0:50020    
      0.0.0.0:*
     
     
  LISTEN    
 17772/java


tcp  
     0
     0
127.0.0.1:9000    
     0.0.0.0:*
     
     
  LISTEN    
 17642/java


tcp  
     0
     0
0.0.0.0:8040    
     
 0.0.0.0:*    
     
    LISTEN  
   17159/java


tcp  
     0
     0
0.0.0.0:50090    
      0.0.0.0:*
     
     
  LISTEN    
 17938/java


tcp  
     0
     0
0.0.0.0:8042    
     
 0.0.0.0:*    
     
    LISTEN  
   17159/java


tcp  
     0
     0
127.0.0.1:41870    
    0.0.0.0:*  
     
      LISTEN
   
 17772/java


tcp  
     0
     0
0.0.0.0:19888
     
    0.0.0.0:*  
     
      LISTEN
   
 17493/java


tcp  
     0
     0
0.0.0.0:10033    
      0.0.0.0:*
     
     
  LISTEN    
 17493/java


tcp  
     0
     0
0.0.0.0:50070
     
    0.0.0.0:*  
     
      LISTEN
   
 17642/java


tcp  
     0
     0
0.0.0.0:8088  
     
   0.0.0.0:*  
     
      LISTEN
   
 17045/java


tcp  
     0
     0
0.0.0.0:50010    
      0.0.0.0:*
     
     
  LISTEN    
 17772/java


tcp  
     0
     0
0.0.0.0:13562    
      0.0.0.0:*
     
     
  LISTEN    
 17159/java


tcp  
     0
     0
0.0.0.0:50075    
      0.0.0.0:*
     
     
  LISTEN    
 17772/java


tcp  
     0
     0
0.0.0.0:8030    
     
 0.0.0.0:*    
     
    LISTEN  
   17045/java


tcp  
     0
     0
0.0.0.0:8031    
     
 0.0.0.0:*    
     
    LISTEN  
   17045/java


tcp  
     0
     0
0.0.0.0:36864    
      0.0.0.0:*
     
     
  LISTEN    
 17159/java


tcp  
     0
     0
0.0.0.0:8032    
     
 0.0.0.0:*    
     
    LISTEN  
   17045/java

















4.ResourceManager
web接口


localhost:8088


Hadoop单节点集群









5.JobHistory Server web接口




localhost:19888

Hadoop单节点集群













							
		





            

            
posted @ 
2016-05-25 22:57 
李庆喜 
阅读(258) 
评论(0) 
编辑 
收藏 
举报