Using Erasure Coding for New Data

You must create a new directory and then set a supported EC policy for the directory.

Create a new directory or choose an existing directory.
View the supported EC policies.

hdfs ec -listPolicies

Enable the supported EC policy.

hdfs ec -enablePolicy -policy <policy>

Set the EC policy for the directory you want to use.

hdfs ec -setPolicy -path <directory> [-policy <policyName>]

path: This is required. Specify the HDFS directory you want to apply the EC policy to.
policy: This is optional. The EC policy you want to use for the directory you specified. If you do not provide this parameter, the EC policy you specified in the fallback Erasure Coding Policy setting from Acceldata Manager is used.

Set the destination for data to the directory you enabled EC for. No action beyond that is required. When the data is written to the directory, it will be erasure coded based on the policy you set.

Info

The supported policies are RS-10-4-1024k , RS-3-2-1024k ,RS-6-3-1024k , RS-LEGACY-6-3-1024k , and XOR-2-1-1024.

You can use the below example for Enabling Erasure Coding:

Check the supported EC policy “hdfs ec -listPolicies“.

[root@odp-krb1 keytabs]# hdfs ec -listPolicies Erasure Coding Policies: ErasureCodingPolicy=[Name=RS-10-4-1024k, Schema=[ECSchema=[Codec=rs, numDataUnits=10, numParityUnits=4]], CellSize=1048576, Id=5], State=DISABLED ErasureCodingPolicy=[Name=RS-3-2-1024k, Schema=[ECSchema=[Codec=rs, numDataUnits=3, numParityUnits=2]], CellSize=1048576, Id=2], State=DISABLED ErasureCodingPolicy=[Name=RS-6-3-1024k, Schema=[ECSchema=[Codec=rs, numDataUnits=6, numParityUnits=3]], CellSize=1048576, Id=1], State=ENABLED ErasureCodingPolicy=[Name=RS-LEGACY-6-3-1024k, Schema=[ECSchema=[Codec=rs-legacy, numDataUnits=6, numParityUnits=3]], CellSize=1048576, Id=3], State=DISABLED ErasureCodingPolicy=[Name=XOR-2-1-1024k, Schema=[ECSchema=[Codec=xor, numDataUnits=2, numParityUnits=1]], CellSize=1048576, Id=4], State=DISABLED

Select the appropriate policy for the use case “dfs ec -enablePolicy -policy XOR-2-1-1024k”.

[root@odp-krb1 keytabs]# hdfs ec -enablePolicy -policy XOR-2-1-1024k Erasure coding policy XOR-2-1-1024k is enabled

Set policy XOR-2-1-1024k on tmp directory “hdfs ec -setPolicy -path /tmp -policy XOR-2-1-1024k”.

[root@odp-krb1 keytabs]# hdfs ec -setPolicy -path /tmp -policy XOR-2-1-1024k Set XOR-2-1-1024k erasure coding policy on /tmp Warning: setting erasure coding policy on a non-empty directory will not automatically convert existing files to XOR-2-1-1024k erasure coding policy

Validate the set policy on the path.

hdfs ec -getPolicy -path /tmp XOR-2-1-1024k

Unset policy on the hdfs directory.

[root@odp-krb1 keytabs]# hdfs ec -unsetPolicy -path /tmp Unset erasure coding policy from /tmp Warning: unsetting erasure coding policy on a non-empty directory will not automatically convert existing files to replicated data.

Disable the set policy on HDFS.

[root@odp-krb1 keytabs]# hdfs ec -disablePolicy -policy XOR-2-1-1024k Erasure coding policy XOR-2-1-1024k is disabled

Last updated on Mar 12, 2025

Was this page helpful?