hadoop配置day01

hadoop

安装jdk

配置文件：
sudo vim /etc/profile
配置文件：
export JAVA_HOME=/home/hadoop/jvm/jdk1.8.0_341
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=${JAVA_HOME}/bin:$PATH
重新加载配置文件：
source /etc/profile
验证是否成功：
java -version

配置ssh免密登录

安装ssh:
sudo ape-get install ssh
产生ssh key:
ssh-keygen -t rsa
查看ssh key:
ll ~/.ssh
将id_rsa.pub放许可证文件:
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
更改权限：
chmod 755~
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
验证是否成功：
ssh -Version
用hostname查看主机名：
ssh ubuntu(主机名)
退出ssh:exit

hadoop安装

1.解压安装包
cd 到想要安装的目录：
tar -zxvf hadoop-3.1.1.tar.gz

hadoop环境变量配置准备

1.查看本机ip:
ifconfig
本机ip为：192.168.232.9
2.查看主机名:
hostname
本机主机名为：nbuntu
3.将ip地址和主机名写进/etc/hosts:
sudo vim /etc/hosts

环境变量配置

1.编辑配置文件
sudo vim ~/.bashrc
export JAVA_HOME=/home/hadoop/jvm/jdk1.8.0_341
export HADOOP_HOME=/home/hadoop01/hadoop-3.1.1
export PATH=$PATH:$HADOOP_HOME/binexport PATH=$PATH:$HADOOP_HOME/sbin
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib :$HADOOP_COMMON_LIB_NATIVE_DIR"
export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native:$JAVA_LIBRARY_PATH
在 /etc/profile添加
export HADOOP_HOME=/home/hadoop01/hadoop-3.1.1
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin
配置好之后，重新加载一下source /etc/profile
source ~/.bashrc
2.检查是否成功
hadoop version

修改hadoop配置文件

1.修改hadoop-env.sh
sudo vim ./hadoop-3.1.1/etc/hadoop/hadoop-env.sh
添加export JAVA_HOME=/home/hadoop/jvm/jdk1.8.0_341
2.在/home/hadoop01/hadoop-3.1.1/etc/hadoop目录下：
修改core-site.xml
sudo vim core-site.xml
添加：
<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://192.168.232.9:9000</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/home/hadoop01/hadoop-3.1.1/dataNode_1_dir</value>
   </property>
</configuration>
3.修改sudo vim yarn-site.xml
<property>
          <name>yarn.resourcemanager.hostname</name>
          <value>192.168.232.9</value>
</property>
<property>
          <name>yarn.nodemanager.aux-services</name>
          <value>mapreduce_shuffle</value>
</property>
4.修改mapred-site.xml
<configuration>
        <property>
                <name>mapreduce.framework.name</name>
                <value>yarn</value>
        </property>
</configuration>
5.修改sudo vim hdfs-site.xml
<configuration>
        <property>
                <name>dfs.replication</name>
                <value>1</value>
        </property>
        <property>
                <name>dfs.namenode.name.dir</name>
                <value>file:///home/hadoop01/hadoop-3.1.1/hadoop_data/hdfs/namenode</value>
        </property>
        <property>
                <name>dfs.datanode.data.dir</name>
                <value>file:///home/hadoop01/hadoop-3.1.1/hadoop_data/hdfs/datanode</value>
        </property>
        <property>
                <name>dfs.http.address</name>
                <value>192.168.232.9:50070</value>
        </property>

</configuration>
修改/etc/profile
sudo vim /etc/profile
export JAVA_HOME=/home/hadoop/jvm/jdk1.8.0_341
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=${JAVA_HOME}/bin:$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

export HADOOP_HOME=/home/hadoop01/hadoop-3.1.1

6.创建并格式化文件系统
mkdir -p /home/hadoop01/hadoop-3.1.1/hadoop_data/hdfs/namenode
mkdir -p /home/hadoop01/hadoop-3.1.1/hadoop_data/hdfs/datanode
mkdir -p /home/hadoop01/hadoop-3.1.1/dataNode_1_dir/datanode
格式化:
hdfs namenode -format

bug

1.如果出现找不到native本地库的情况，需要在hadoop-env.sh里面加入native的路径，然后重新格式化一下namenode，即可成功。
2.不要轻易格式化namenode，关机需要先关所有进程，避免下次丢失。
结点出不来就格式化，先删文件夹再格式化，要删干净

3.修改ip：sudo vim /etc/netplan/01-network-manager-all.yaml
network:
  ethernets:
    ens33:
      addresses: [192.168.232.1/24]
      gateway4: 192.168.232.2
      nameservers: 
          addresses: [114.114.114.114,8.8.8.8]
  version: 2
  renderer: NetworkManager
生效：sudo netplan apply

修改主机名：sudo vim /etc/hostname

然后解析主机：vim /etc/resolv.conf
nameserver 114.114.114.114
nameserver 8.8.8.8

修改完主机名需要改一下:
sudo vim /etc/hosts
127.0.0.1 localhost
ip 主机名

ping不通百度