侧边栏

Ubuntu kylin优麒麟下配置Hadoop环境

Ubuntu kylin优麒麟下配置Hadoop环境

查看JDK目录

cd /usr/lib/jvm/java-8-openjdk-amd64

查看Hadoop目录

cd /usr/local/hadoop

查看IP地址

ifconfig

ssh服务开启（如果没有开启）

Linux系统的ssh要打开，不然后面连接不上HDFS。

1.问题：

ssh连接时报以下错误：

$ ssh root@192.168.199.22
root@192.168.199.22's password: 
Permission denied, please try again.

2.原因：

系统默认禁止root用户登录ssh

3. 解决：

（1）修改/etc/ssh/sshd_config文件
vim /etc/ssh/sshd_config

PermitRootLogin without-password

改为

PermitRootLogin yes
（2）重启ssh
sudo service ssh restart

Hadoop配置

查看Hadoop的etc文件

cd /usr/local/hadoop/etc/hadoop/

配置$HADOOP_HOME/etc/hadoop下的hadoop-env.sh文件 

vim hadoop-env.sh 

# The java implementation to use.
#export JAVA_HOME=${JAVA_HOME}
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop

配置$HADOOP_HOME/etc/hadoop下的core-site.xml文件

vim core-site.xml
注意IP地址与路径<value>/home/cai/simple/soft/hadoop-2.7.1/tmp</value>
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
<!-- HDFS file path -->
<property>
  <name>fs.default.name</name>
  <value>hdfs://192.168.1.248:9000</value>
 </property>

<property>
  <name>fs.defaultFS</name>
  <value>hdfs://192.168.1.248:9000</value>
 </property>

<property>
  <name>fs.defaultFS</name>
  <value>hdfs://192.168.1.248:9000</value>
 </property>

 <property>
  <name>io.file.buffer.size</name>
  <value>131072</value>
 </property>

 <property>
  <name>hadoop.tmp.dir</name>
  <value>/usr/local/hadoop/tmp</value>
  <description>Abasefor other temporary directories.</description>
 </property>


</configuration>

配置$HADOOP_HOME/etc/hadoop下的hdfs-site.xml文件

vim hdfs-site.xml

这里需要注意的是：如果找不到 /hdfs 文件，可以把文件路径改为 /tmp/dfs 下查找name与data文件
<value>/home/cai/simple/soft/hadoop-2.7.1/hdfs/name</value>
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->
<configuration>
  <property>
   <name>dfs.namenode.name.dir</name>
   <value>/usr/local/hadoop/tmp/dfs/name</value>
 </property>

 <property>
  <name>dfs.datanode.data.dir</name>
    <value>/usr/local/hadoop/tmp/dfs/data</value>
  </property>
<!--
  <property>
   <name>dfs.namenode.name.dir</name>
   <value>/home/cai/simple/soft/hadoop-2.7.1/etc/hadoop
/hdfs/name</value>
 </property>

 <property>
  <name>dfs.datanode.data.dir</name>
    <value>/home/cai/simple/soft/hadoop-2.7.1/etc/hadoop
/hdfs/data</value>
  </property>
-->

 <property>
  <name>dfs.replication</name>
  <value>1</value>
 </property>

 <property>
  <name>dfs.webhdfs.enabled</name>
  <value>true</value>
 </property>


</configuration>

配置$HADOOP_HOME/etc/hadoop下的mapred-site.xml文件

vim mapred-site.xml
如果没有mapred-site.xml，需要自己新创建一个mapred-site.xml。
cp mapred-site.xml.template mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
  <property>
   <name>mapreduce.framework.name</name>
   <value>yarn</value>
 </property>

 <property>
  <name>mapreduce.jobhistory.address</name>
  <value>192.168.1.248:10020</value>
 </property>

 <property>
  <name>mapreduce.jobhistory.webapp.address</name>
  <value>192.168.1.248:19888</value>
 </property>

</configuration>

配置$HADOOP_HOME/etc/hadoop下的yarn-site.xml文件

vim yarn-site.xml

<?xml version="1.0"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->


<configuration>

<!-- Site specific YARN configuration properties -->
<property>
   <name>yarn.nodemanager.aux-services</name>
   <value>mapreduce_shuffle</value>
  </property>

  <property>
   <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
   <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>

 <property>
   <name>yarn.resourcemanager.address</name>
   <value>192.168.1.248:8032</value>
  </property>

<property>
   <name>yarn.resourcemanager.scheduler.address</name>
   <value>192.168.1.248:8030</value>
  </property>

  <property>
   <name>yarn.resourcemanager.resource-tracker.address</name>
   <value>192.168.1.248:8035</value>
  </property>

 <property>
   <name>yarn.resourcemanager.admin.address</name>
   <value>192.168.1.248:8033</value>
  </property>

  <property>
   <name>yarn.resourcemanager.webapp.address</name>
   <value>192.168.1.248:8088</value>
  </property>

</configuration>

配置/etc/profile文件

vim /etc/profile

# /etc/profile: system-wide .profile file for the Bourne shell (sh(1))
# and Bourne compatible shells (bash(1), ksh(1), ash(1), ...).

if [ "$PS1" ]; then
  if [ "$BASH" ] && [ "$BASH" != "/bin/sh" ]; then
    # The file bash.bashrc already sets the default PS1.
    # PS1='\h:\w\$ '
    if [ -f /etc/bash.bashrc ]; then
      . /etc/bash.bashrc
    fi
  else
    if [ "`id -u`" -eq 0 ]; then
      PS1='# '
    else
      PS1='$ '
    fi
  fi
fi

if [ -d /etc/profile.d ]; then
  for i in /etc/profile.d/*.sh; do
    if [ -r $i ]; then
      . $i
    fi
  done
  unset i
fi

#java environment
  export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
  export CLASSPATH=.:${JAVA_HOME}/jre/lib/rt.jar:${JAVA_HOME}/lib/dt.jar:${JAVA_HOME}/lib/tools.jar
  export PATH=$PATH:${JAVA_HOME}/bin


  export HADOOP_HOME=/usr/local/hadoop
  export PATH=$PATH:${JAVA_HOME}/bin:${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin

更新配置文件

让配置文件生效，需要执行命令source /etc/profile

source /etc/profile

格式化NameNode

格式化NameNode，在任意目录下执行 hdfs namenode -format 或者 hadoop namenode -format ，实现格式化。

hdfs namenode -format 
或者
hadoop namenode -format

启动Hadoop集群

启动Hadoop进程，首先执行命令start-dfs.sh,启动HDFS系统。

start-dfs.sh

启动yarn集群

start-yarn.sh

jps查看配置信息

jps

UI测试

测试HDFS和yarn（推荐使用火狐浏览器）有两种方法，一个是在命令行中打开，另一个是直接双击打开

firefox

端口:8088与50070端口

首先在浏览器中输入http://172.16.12.37:50070/  (HDFS管理界面）（此IP是自己虚拟机的IP地址，端口为固定端口）每个人的IP不一样，根据自己的IP地址来定。。。

在浏览器中输入http://172.16.12.37:8088/（MR管理界面）（此IP是自己虚拟机的IP地址，端口为固定端口）每个人的IP不一样，根据自己的IP地址来定。。。

posted @ 2020-10-23 17:41  菜鸟-传奇  阅读(612)  评论(0)  编辑  收藏  举报