【hadoop集群】一、安装hadoop3.1.3 核心配置文件内容以及可能出现的问题

1. INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:41576. Already tried 0 time(s); 

或者卡在

2022-01-04 01:26:35,209 INFO mapreduce.Job: Running job: job_1641277332916_0002，后面跑不下去了

解决办法：vim /etc/hosts 

把127.0.0.1这行注释掉

 

2、hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar wordcount /input /output

Got exception: java.net.ConnectException: Call From localhost/127.0.0.1 to localhost:43696 failed on connection exception: java.net.ConnectException: Connection refused;

解决办法：vim /etc/hosts 

把127.0.0.1这行注释掉

 

3、bin/beeline -u jdbc:hive2://hadoop103:10000/metastore -n zhyx

FAILED: HiveException java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
解决办法换jar包，hive包与hadoop冲突，需要编译

 

5、Error: Could not open client transport with JDBC Uri: jdbc:hive2://hadoop02:10000: Failed to open new session: java.net.ConnectException: Call From hadoop02/192.168.29.71 to hadoop01:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused (state=08S01,code=0)

把配置文件的主机节点设置为同hive同一节点

 

4、WARN jdbc.HiveConnection: Failed to connect to hadoop103:10000

Could not open connection to the HS2 server. Please check the server URI and if the URI is correct, then ask the administrator to check the server status.

Error: Could not open client transport with JDBC Uri: jdbc:hive2://hadoop103:10000/metastore: Failed to open new session: java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: zhyx is not allowed to impersonate zhyx (state=08S01,code=0)

 

netstat -nlpt |grep 9083 查看这个进程是否开启

netstat -nlpt |grep 10000  查看这个进程是否开启

 

查看namenode datanode占用内存：

jmap -heap  2611（namenode的进程号）

 

 

 

权限不够：增加配置  

zhyx是用户名，用户组

<property>
    <name>hadoop.proxyuser.zhyx.hosts</name>
    <value>*</value>
</property>
<property>
    <name>hadoop.proxyuser.zhyx.groups</name>
    <value>*</value>
</property>

　　

 

 

 

hadoop四个配置文件

 

<!-- core指定 NameNode 的地址 -->
 <property>
 <name>fs.defaultFS</name>
 <value>hdfs://hadoop101:8020</value>
 </property>
 <!-- 指定 hadoop 数据的存储目录 -->
 <property>
 <name>hadoop.tmp.dir</name>
 <value>/opt/module/hadoop-3.1.3/data</value>
 </property>
 <!-- 配置 HDFS 网页登录使用的静态用户为 zhyx -->
 <property>
 <name>hadoop.http.staticuser.user</name>
 <value>zhyx</value>
 </property>



<property>
    <name>hadoop.proxyuser.zhyx.hosts</name>
    <value>*</value>
</property>
<property>
    <name>hadoop.proxyuser.zhyx.groups</name>
    <value>*</value>
</property>



<!-- hdfs nn web 端访问地址-->
<property>
 <name>dfs.namenode.http-address</name>
 <value>hadoop101:9870</value>
 </property>
<!-- 2nn web 端访问地址-->
 <property>
 <name>dfs.namenode.secondary.http-address</name>
 <value>hadoop103:9868</value>
 </property>





<!-- yarn指定 MR 走 shuffle -->
 <property>
 <name>yarn.nodemanager.aux-services</name>
 <value>mapreduce_shuffle</value>
 </property>
 <!-- 指定 ResourceManager 的地址-->
 <property>
 <name>yarn.resourcemanager.hostname</name>
 <value>hadoop102</value>
 </property>
 <!-- 环境变量的继承 -->
 <property>
 <name>yarn.nodemanager.env-whitelist</name>
<value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
 </property>

<!-- 开启日志聚集功能 -->
<property>
 <name>yarn.log-aggregation-enable</name>
 <value>true</value>
</property>
<!-- 设置日志聚集服务器地址 -->
<property> 
 <name>yarn.log.server.url</name> 
 <value>http://hadoop103:19888/jobhistory/logs</value>
</property>
<!-- 设置日志保留时间为 7 天 -->
<property>
 <name>yarn.log-aggregation.retain-seconds</name>
 <value>604800</value>
</property>

<!-- 添加tez避免虚拟机检查 -->
<property>
 <name>yarn.nodemanager.vmem-check-enabled</name>
 <value>false</value>
</property>

 

 

 

<!-- 指定 MapReduce 程序运行在 Yarn 上 -->
 <property>
 <name>mapreduce.framework.name</name>
 <value>yarn</value>
 </property>

<!-- 历史服务器端地址 -->
<property>
 <name>mapreduce.jobhistory.address</name>
 <value>hadoop103:10020</value>
</property>
<!-- 历史服务器 web 端地址 -->
<property>
 <name>mapreduce.jobhistory.webapp.address</name>
 <value>hadoop103:19888</value>
</property>


start-dfs.sh
start-yarn.sh
mapred --daemon start historyserver

mapred --daemon stop historyserver

hadoop jar /opt/module/hadoop-3.1.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar wordcount /input/ /output


hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar wordcount /input /output

 Got exception: java.net.ConnectException: Call From localhost/127.0.0.1 to localhost:43696 failed on connection exception: java.net.ConnectException: Connection refused;


-X)0yn6k2ZR6



<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
 <!-- jdbc 连接的 URL -->
 <property>
 <name>javax.jdo.option.ConnectionURL</name>
 <value>jdbc:mysql://hadoop103:3306/metastore?useSSL=false</value>
</property>
 <!-- jdbc 连接的 Driver-->
 <property>
 <name>javax.jdo.option.ConnectionDriverName</name>
 <value>com.mysql.jdbc.Driver</value>
</property>
<!-- jdbc 连接的 username-->
 <property>
 <name>javax.jdo.option.ConnectionUserName</name>
 <value>root</value>
 </property>
 <!-- jdbc 连接的 password -->
 <property>
 <name>javax.jdo.option.ConnectionPassword</name>
 <value>zhyx</value>
</property>
 <!-- Hive 元数据存储版本的验证 -->
 <property>
 <name>hive.metastore.schema.verification</name>
 <value>false</value>
</property>
 <!--元数据存储授权-->
 <property>
 <name>hive.metastore.event.db.notification.api.auth</name>
 <value>false</value>
 </property>
 <!-- Hive 默认在 HDFS 的工作目录 -->
 <property>
<name>hive.metastore.warehouse.dir</name>
 <value>/user/hive/warehouse</value>
 </property>

<!-- 指定存储元数据要连接的地址 -->
 <property>
 <name>hive.metastore.uris</name>
 <value>thrift://hadoop103:9083</value>
 </property>

<!-- 指定 hiveserver2 连接的 host -->
 <property>
 <name>hive.server2.thrift.bind.host</name>
<value>hadoop103</value>
 </property>
 <!-- 指定 hiveserver2 连接的端口号 -->
 <property>
 <name>hive.server2.thrift.port</name>
 <value>10000</value>
 </property>

</configuration>