Hadoop添加自定义纠删码

Hadoop3.x支持了纠删码策略
增添自定义纠删码策略示例：
Hadoop存在一个 <HDFS客户端安装目录>/HDFS/hadoop/etc/hadoop/user_ec_policies.xml.template 的EC策略的XML文件示例。

1. 按照模板自定义纠删码策略，示例如下，自定义了两个纠删码策略XORk2m1和RS-legacyk12m4策略：

<?xml version="1.0"?> 
 <configuration> 
 <!-- The version of EC policy XML file format, it must be an integer --> 
 <layoutversion>1</layoutversion> 
 <schemas> 
   <!-- schema id is only used to reference internally in this document --> 
   <schema id="XORk2m1"> 
     <!-- The combination of codec, k, m and options as the schema ID, defines 
      a unique schema, for example 'xor-2-1'. schema ID is case insensitive --> 
     <!-- codec with this specific name should exist already in this system --> 
     <codec>xor</codec> 
     <k>2</k> 
     <m>1</m> 
     <options> </options> 
   </schema> 
   <schema id="RS-legacyk12m4"> 
     <codec>rs-legacy</codec> 
     <k>12</k> 
     <m>4</m> 
     <options> </options> 
   </schema> 
 </schemas> 
 <policies> 
   <policy> 
     <!-- the combination of schema ID and cellsize(in unit k) defines a unique 
      policy, for example 'xor-2-1-256k', case insensitive --> 
     <!-- schema is referred by its id --> 
     <schema>XORk2m1</schema> 
     <!-- cellsize must be an positive integer multiple of 1024(1k) --> 
     <!-- maximum cellsize is defined by 'dfs.namenode.ec.policies.max.cellsize' property --> 
     <cellsize>131072</cellsize> 
   </policy> 
   <policy> 
     <schema>RS-legacyk12m4</schema> 
     <cellsize>262144</cellsize> 
   </policy> 
 </policies> 
 </configuration>

2. 将上面的纠删码策略加入Hadoop

hdfs ec -addPolicies -policyFile <xmlLocation>

3. 查看是否加入自定义policy

hdfs ec -listPolicies

4. 默认加入的策略是disable状态，所以需要进行enable

hdfs ec -enablePolicy -policy <policyname>

以上便完成了添加自定义纠删码！
tips：hdfs ec -h 查看hdfs ec相关命令详情