(实验性质的) hadoop fsck健康检查副本异常信息Target Replicas is 3 but found 2 live replica(s)
之前下架过节点,导致副本数量不全,再做健康检查时,发现许多这样的信息。
当然这样的信息并不是一定有问题,比如以下目录副本数就只有一个;
执行的flink的任务时,/flink/job/目录
hbase的一些tmp临时目录 /hbase/tmp
上传的jar包程序 /jars/
Under replicated BP-1555553207-10.0.50.200-1625229209582:blk_1144828808_71104208. Target Replicas is 3 but found 2 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
以下为百度出来的修复方法,有待验证:
1. 获取待修复的副本清单
nohup hadoop fsck /user/hive/warehouse | grep 'Under replicated' | awk -F':' '{print $1}' >> ./under_replicated_files 2>&1 &
2.排除掉/user/hive/warehouse/ods.db/目录
sed -e '//user/hive/warehouse/ods.db//d' under_replicated_files > miss_replicated_files
3.遍历并修复副本
for hdfsfile in `cat miss_replicated_files`;
do echo "Fixing $hdfsfile :" ;
hadoop fs -setrep 3 $hdfsfile;
done
本文来自博客园,作者:硅谷工具人,转载请注明原文链接:https://www.cnblogs.com/30go/p/16941700.html