Using Erasure Coding for New Data

You must create a new directory and then set a supported EC policy for the directory.

Create a new directory or choose an existing directory.
View the supported EC policies.

Bash
    
 
hdfs ec -listPolicies
Copy

Enable the supported EC policy.

Bash
    
 
hdfs ec -enablePolicy -policy <policy>
Copy

Set the EC policy for the directory you want to use.

Bash
    
 
hdfs ec -setPolicy -path <directory> [-policy <policyName>]
Copy

path: This is required. Specify the HDFS directory you want to apply the EC policy to.
policy: This is optional. The EC policy you want to use for the directory you specified. If you do not provide this parameter, the EC policy you specified in the fallback Erasure Coding Policy setting from Acceldata Manager is used.

Set the destination for data to the directory you enabled EC for. No action beyond that is required. When the data is written to the directory, it will be erasure coded based on the policy you set.

The supported policies are RS-10-4-1024k , RS-3-2-1024k ,RS-6-3-1024k , RS-LEGACY-6-3-1024k , and XOR-2-1-1024.

You can use the below example for Enabling Erasure Coding:

Check the supported EC policy “hdfs ec -listPolicies“.

Bash
    
[root@odp-krb1 keytabs]# hdfs ec -listPoliciesErasure Coding Policies:ErasureCodingPolicy=[Name=RS-10-4-1024k, Schema=[ECSchema=[Codec=rs, numDataUnits=10, numParityUnits=4]], CellSize=1048576, Id=5], State=DISABLEDErasureCodingPolicy=[Name=RS-3-2-1024k, Schema=[ECSchema=[Codec=rs, numDataUnits=3, numParityUnits=2]], CellSize=1048576, Id=2], State=DISABLEDErasureCodingPolicy=[Name=RS-6-3-1024k, Schema=[ECSchema=[Codec=rs, numDataUnits=6, numParityUnits=3]], CellSize=1048576, Id=1], State=ENABLEDErasureCodingPolicy=[Name=RS-LEGACY-6-3-1024k, Schema=[ECSchema=[Codec=rs-legacy, numDataUnits=6, numParityUnits=3]], CellSize=1048576, Id=3], State=DISABLEDErasureCodingPolicy=[Name=XOR-2-1-1024k, Schema=[ECSchema=[Codec=xor, numDataUnits=2, numParityUnits=1]], CellSize=1048576, Id=4], State=DISABLED
Copy

Select the appropriate policy for the use case “dfs ec -enablePolicy -policy XOR-2-1-1024k”.

Bash
    
​x
 
[root@odp-krb1 keytabs]# hdfs ec -enablePolicy -policy XOR-2-1-1024k​Erasure coding policy XOR-2-1-1024k  is enabled
Copy

Set policy XOR-2-1-1024k on tmp directory “hdfs ec -setPolicy -path /tmp -policy XOR-2-1-1024k”.

Bash
    
[root@odp-krb1 keytabs]# hdfs ec -setPolicy -path /tmp  -policy XOR-2-1-1024k​Set XOR-2-1-1024k erasure coding policy on /tmp​Warning: setting erasure coding policy on a non-empty directory will not automatically convert existing files to XOR-2-1-1024k erasure coding policy
Copy

Validate the set policy on the path.

Bash
    
 
hdfs ec -getPolicy -path /tmp​XOR-2-1-1024k
Copy

Unset policy on the hdfs directory.

Bash
    
[root@odp-krb1 keytabs]# hdfs ec -unsetPolicy -path /tmpUnset erasure coding policy from /tmpWarning: unsetting erasure coding policy on a non-empty directory will not automatically convert existing files to replicated data.
Copy

Disable the set policy on HDFS.

Bash
    
 
[root@odp-krb1 keytabs]# hdfs ec -disablePolicy -policy XOR-2-1-1024k​Erasure coding policy XOR-2-1-1024k is disabled
Copy

Last updated on Mar 10, 2025

Was this page helpful?