hdfs 重启后报错

重启之后发现namenode报错了，查看日志

2018-01-16 18:30:06,923 WARN org.apache.hadoop.ipc.Client: Couldn't setup connection for hdfs/bigdatanode02.csebank.com@CSEBANK.COM to bigdatanode02.csebank.com/170.100.100.96:8485
org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:378)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:594)
at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:396)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:761)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:757)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:756)
at org.apache.hadoop.ipc.Client$Connection.access$3000(Client.java:396)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1557)
at org.apache.hadoop.ipc.Client.call(Client.java:1480)
at org.apache.hadoop.ipc.Client.call(Client.java:1441)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
at com.sun.proxy.$Proxy14.getEditLogManifest(Unknown Source)
at org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolTranslatorPB.getEditLogManifest(QJournalProtocolTranslatorPB.java:245)
at org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannel$13.call(IPCLoggerChannel.java:556)
at org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannel$13.call(IPCLoggerChannel.java:553)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

 

 

查看日志发现 keytab问题，立马去重新生成一个，试验后还是同样的错误。

查看相关信息发现，密钥周期没问题，也可以验证成功，时间和时区也都没问题。 最后查找日志发现 JAVA_HOME 不对，我的java目录下有两个版本的java ！！！！ 不知道谁加进去的.......

然后日志文件出现了一行这种信息

ls -rvd /usr/java/jdk1.7.0_67 /usr/java/jdk1.7.0_69

export JAVA_HOME=/usr/java/jdk1.7.0_69

我的JAVA_HOME 明明设置的是/usr/java/jdk1.7.0_67 ，我怀疑是这个问题。于是将jdk1.7.0_69移出目录。

果然，成功了！！