查找HDFS小文件数量

查找HDFS有哪些小文件以及统计数量

hdfs 导出fsimage文件

hdfs dfsadmin -fetchImage  /data

转换为可视化数据

hdfs oiv -i /data/fsimage_0000000000930647029 -o /data/fsimage.csv -p Delimited  -delimiter ","

删除数据第一行标题

$ sed -i 1d /data/fsimage.csv

把数据导入mysql

CREATE TABLE
IF
    NOT EXISTS `images_info` (
  `Path` VARCHAR ( 100 ),
  `Replication` INT ( 20 ),
  `ModificationTime` VARCHAR ( 100 ),
  `AccessTime` VARCHAR ( 100 ),
  `PreferredBlockSize` BIGINT ( 100 ),
  `BlocksCount` INT ( 20 ),
  `FileSize` BIGINT ( 100 ),
  `NSQUOTA` VARCHAR ( 100 ),
  `DSQUOTA` VARCHAR ( 100 ),
  `Permission` VARCHAR ( 100 ),
  `UserName` VARCHAR ( 100 ),
    `GroupName` VARCHAR ( 100 ) 
    ) CHARSET = utf8

加载数据到mysql

load data local infile '/data/fsimage.csv' into table imgaes_info;

查找小文件

select count(Path), count(FileSize) from images_info where FileSize < 500;