Hadoop重启练习：namenode running as process 4635. Stop it first.
重启记录，hadoop练习。
先熟悉一下语句：
[root@bigdata hadoop]# $HADOOP_HOME/sbin/start-all.sh This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh Starting namenodes on [bigdata] bigdata: namenode running as process 4635. Stop it first. bigdata: datanode running as process 4818. Stop it first. Starting secondary namenodes [0.0.0.0] 0.0.0.0: secondarynamenode running as process 5090. Stop it first. starting yarn daemons resourcemanager running as process 5300. Stop it first. bigdata: nodemanager running as process 5435. Stop it first.
[root@bigdata hadoop]# $HADOOP_HOME/sbin/stop-all.sh This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh Stopping namenodes on [bigdata] bigdata: stopping namenode bigdata: stopping datanode Stopping secondary namenodes [0.0.0.0] 0.0.0.0: stopping secondarynamenode stopping yarn daemons stopping resourcemanager bigdata: stopping nodemanager bigdata: nodemanager did not stop gracefully after 5 seconds: killing with kill -9 no proxyserver to stop
[root@bigdata hadoop]# $HADOOP_HOME/bin/hdfs namenode -format
。。。略
20/04/13 21:08:39 INFO common.Storage: Storage directory /home/admin/Downloads/hadoop-2.10.0/current/dfs/name has been successfully formatted.
20/04/13 21:08:39 INFO namenode.FSImageFormatProtobuf: Saving image file /home/admin/Downloads/hadoop-2.10.0/current/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
20/04/13 21:08:40 INFO namenode.FSImageFormatProtobuf: Image file /home/admin/Downloads/hadoop-2.10.0/current/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 318 bytes saved in 0 seconds .
20/04/13 21:08:40 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
20/04/13 21:08:40 INFO namenode.FSImage: FSImageSaver clean checkpoint: txid = 0 when meet shutdown.
20/04/13 21:08:40 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at bigdata/192.168.0.108
************************************************************/
开始重启：
[root@bigdata hadoop]# $HADOOP_HOME/sbin/start-all.sh This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh Starting namenodes on [bigdata] bigdata: starting namenode, logging to /home/admin/Downloads/hadoop-2.10.0/logs/hadoop-root-namenode-bigdata.out bigdata: starting datanode, logging to /home/admin/Downloads/hadoop-2.10.0/logs/hadoop-root-datanode-bigdata.out Starting secondary namenodes [0.0.0.0] 0.0.0.0: starting secondarynamenode, logging to /home/admin/Downloads/hadoop-2.10.0/logs/hadoop-root-secondarynamenode-bigdata.out starting yarn daemons starting resourcemanager, logging to /home/admin/Downloads/hadoop-2.10.0/logs/yarn-root-resourcemanager-bigdata.out bigdata: starting nodemanager, logging to /home/admin/Downloads/hadoop-2.10.0/logs/yarn-root-nodemanager-bigdata.out
jps检查：
[root@bigdata hadoop]# jps 117680 ResourceManager 117815 NodeManager 5976 JobHistoryServer 118456 Jps 117243 DataNode 117486 SecondaryNameNode
（！！NameNode没有起来！！）
http://xxx.xxx.xxx.xxx:50070/ 能正常访问
File system下面也都清空。
但是Namenode没有跑起来。
重启以后，查看namenode和dataNode的VERSION文件。
cd $HADOOP_HOME/current/dfs/name/current
$HADOOP_HOME/current/dfs/name/current/VERSION 内容如下：
[root@bigdata current]# cat VERSION
#Mon Apr 13 21:08:39 GMT+08:00 2020
namespaceID=1944953128
clusterID=CID-ae34d376-3b62-420e-8740-d44ceb892e85
cTime=1586783319430
storageType=NAME_NODE
blockpoolID=BP-755950590-192.168.0.108-1586783319430
layoutVersion=-63
发现两个文件的clusterID不一致。
[root@bigdata current]# pwd
/home/admin/Downloads/hadoop-2.10.0/current/data/current
[root@bigdata current]# cat VERSION
#Sat Apr 11 16:56:37 CST 2020
storageID=DS-a455c69f-f043-4e5d-98ec-56ab0b1be319
clusterID=CID-f25e7b2a-fad2-4d14-b7c6-18976875b42c
cTime=0
datanodeUuid=230a4d76-08f2-49a3-96fc-9c1217ca9693
storageType=DATA_NODE
layoutVersion=-57
参考网上的办法：
不知道这个log是不是和这个VERSION里的clusterID有关。但是我把data VERSION的ClusterID改成 CID-ae34d376-3b62-420e-8740-d44ceb892e85 后，照样执行 sqoop import报错。
2020-04-13 21:14:19,687 WARN org.apache.hadoop.hdfs.server.common.Storage: Failed to add storage directory [DISK]file:/home/admin/Downloads/hadoop-2.10.0/current/data/ java.io.IOException: Incompatible clusterIDs in /home/admin/Downloads/hadoop-2.10.0/current/data: namenode clusterID = CID-ae34d376-3b62-420e-8740-d44ceb892e85; datanode clusterID = CID-f25e7b2a-fad2-4d14-b7c6-18976875b42c
2020-04-13 22:22:39,373 WARN org.apache.hadoop.hdfs.server.common.Storage: Failed to add storage directory [DISK]file:/home/admin/Downloads/hadoop-2.10.0/current/data/ org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /home/admin/Downloads/hadoop-2.10.0/current/data is in an inconsistent state: file VERSION has cTime missing.
结果：失败。
之后我把VERSION文件删了。这肯定不行。
重新stop、reform、start了一遍。
发现Namenode起来了，Datanode没起来。
查看datanode日志。说current directory is not empty。我就像把/current/data下面东西删了。
org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /home/admin/Downloads/hadoop-2.10.0/current/data is in an inconsistent state: Can't format the storage directory because the current directory is not empty.
把..../current/data/current删了。 删之前备份了一下。
cd /home/admin/Downloads/hadoop-2.10.0/current/data
[root@bigdata data]# rm -rf current
然后单独启动了一下 datanode.
[root@bigdata data]# $HADOOP_HOME/sbin/hadoop-daemons.sh --config /home/admin/Downloads/hadoop-2.10.0/etc/hadoop start datanode bigdata: starting datanode, logging to /home/admin/Downloads/hadoop-2.10.0/logs/hadoop-root-datanode-bigdata.out [root@bigdata data]# jps 704 DataNode 130083 ResourceManager 129426 NameNode 853 Jps 5976 JobHistoryServer 130234 NodeManager 129869 SecondaryNameNode
看了一下，又重新生成了新的.../current/data/current 文件
再次执行，没有报错。重启成功！！！
sqoop list-tables --connect jdbc:mysql://localhost:3306/sqooptest --username root --password admin
cd /home/admin/Downloads/hadoop-2.10.0/current/data/current/
[root@bigdata current]# cat VERSION #Mon Apr 13 22:43:03 GMT+08:00 2020 storageID=DS-7a957477-8bc1-4d3a-bca3-a3018860b2cf clusterID=CID-18a2dabf-9a54-4185-94df-fe3998aa4454 cTime=0 datanodeUuid=0d3febb0-45cc-4dae-b637-aa152be646be storageType=DATA_NODE layoutVersion=-57
[root@bigdata current]# cat $HADOOP_HOME/current/dfs/name/current/VERSION #Mon Apr 13 22:32:19 GMT+08:00 2020 namespaceID=1501710775 clusterID=CID-18a2dabf-9a54-4185-94df-fe3998aa4454 cTime=1586788339332 storageType=NAME_NODE blockpoolID=BP-902172903-192.168.0.108-1586788339332 layoutVersion=-63
比较一下, 确实clusterID都是同一个。
ref：https://blog.csdn.net/xiaozelulu/article/details/80386771
======================
其他：
对于hbase启动顺序：hadoop-->zookeeper-->hbase
关闭顺序：hbase-->zookeeper-->hadoop
执行 $HADOOP_HOME/bin/hdfs namenode -format 的时候，有一行提示：
Re-format filesystem in Storage Directory /home/admin/Downloads/hadoop-2.10.0/current/dfs/name ? (Y or N) Y
20/04/13 21:08:39 INFO common.Storage: Will remove files: xxxxxxxxxx
就把$HADOOP_HOME/current/dfs/name/current 下的文件清空了。
没有清空前，我们可以看一下有什么？
cd $HADOOP_HOME/current/dfs/name/current
（里面有太多文件，几百个，我截取了一部分）
[root@bigdata current]# ll -a total 7088 drwxr-xr-x. 2 root root 94208 Apr 13 20:46 . drwxr-xr-x. 3 root root 21 Apr 13 20:46 .. -rw-r--r--. 1 root root 42 Apr 11 17:58 edits_0000000000000000014-0000000000000000015 -rw-r--r--. 1 root root 42 Apr 11 17:59 edits_0000000000000000016-0000000000000000017 -rw-r--r--. 1 root root 42 Apr 11 18:00 edits_0000000000000000018-0000000000000000019 -rw-r--r--. 1 root root 42 Apr 11 18:01 edits_0000000000000000020-0000000000000000021 -rw-r--r--. 1 root root 42 Apr 11 18:02 edits_0000000000000000022-0000000000000000023 -rw-r--r--. 1 root root 42 Apr 13 20:32 edits_0000000000000006483-0000000000000006484 -rw-r--r--. 1 root root 42 Apr 13 20:33 edits_0000000000000006485-0000000000000006486 -rw-r--r--. 1 root root 42 Apr 13 20:34 edits_0000000000000006487-0000000000000006488 -rw-r--r--. 1 root root 42 Apr 13 20:35 edits_0000000000000006489-0000000000000006490 -rw-r--r--. 1 root root 42 Apr 13 20:36 edits_0000000000000006491-0000000000000006492 -rw-r--r--. 1 root root 42 Apr 13 20:37 edits_0000000000000006493-0000000000000006494 -rw-r--r--. 1 root root 42 Apr 13 20:38 edits_0000000000000006495-0000000000000006496 -rw-r--r--. 1 root root 42 Apr 13 20:39 edits_0000000000000006497-0000000000000006498 -rw-r--r--. 1 root root 42 Apr 13 20:40 edits_0000000000000006499-0000000000000006500 -rw-r--r--. 1 root root 42 Apr 13 20:41 edits_0000000000000006501-0000000000000006502 -rw-r--r--. 1 root root 42 Apr 13 20:42 edits_0000000000000006503-0000000000000006504 -rw-r--r--. 1 root root 42 Apr 13 20:43 edits_0000000000000006505-0000000000000006506 -rw-r--r--. 1 root root 42 Apr 13 20:44 edits_0000000000000006507-0000000000000006508 -rw-r--r--. 1 root root 1048576 Apr 13 20:44 edits_0000000000000006509-0000000000000006509 -rw-r--r--. 1 root root 318 Apr 11 17:24 fsimage_0000000000000000000 -rw-r--r--. 1 root root 62 Apr 11 17:24 fsimage_0000000000000000000.md5 -rw-r--r--. 1 root root 2 Apr 11 17:24 seen_txid -rw-r--r--. 1 root root 217 Apr 11 17:24 VERSION
