Using Erasure Coding for New Data
You must create a new directory and then set a supported EC policy for the directory.
- Create a new directory or choose an existing directory.
- View the supported EC policies.
hdfs ec -listPolicies- Enable the supported EC policy.
hdfs ec -enablePolicy -policy <policy>- Set the EC policy for the directory you want to use.
hdfs ec -setPolicy -path <directory> [-policy <policyName>]- path: This is required. Specify the HDFS directory you want to apply the EC policy to.
- policy: This is optional. The EC policy you want to use for the directory you specified. If you do not provide this parameter, the EC policy you specified in the fallback Erasure Coding Policy setting from Acceldata Manager is used.
- Set the destination for data to the directory you enabled EC for. No action beyond that is required. When the data is written to the directory, it will be erasure coded based on the policy you set.
The supported policies are RS-10-4-1024k , RS-3-2-1024k ,RS-6-3-1024k , RS-LEGACY-6-3-1024k , and XOR-2-1-1024.
You can use the below example for Enabling Erasure Coding:
- Check the supported EC policy “
hdfs ec -listPolicies“.
[root@odp-krb1 keytabs]# hdfs ec -listPoliciesErasure Coding Policies:ErasureCodingPolicy=[Name=RS-10-4-1024k, Schema=[ECSchema=[Codec=rs, numDataUnits=10, numParityUnits=4]], CellSize=1048576, Id=5], State=DISABLEDErasureCodingPolicy=[Name=RS-3-2-1024k, Schema=[ECSchema=[Codec=rs, numDataUnits=3, numParityUnits=2]], CellSize=1048576, Id=2], State=DISABLEDErasureCodingPolicy=[Name=RS-6-3-1024k, Schema=[ECSchema=[Codec=rs, numDataUnits=6, numParityUnits=3]], CellSize=1048576, Id=1], State=ENABLEDErasureCodingPolicy=[Name=RS-LEGACY-6-3-1024k, Schema=[ECSchema=[Codec=rs-legacy, numDataUnits=6, numParityUnits=3]], CellSize=1048576, Id=3], State=DISABLEDErasureCodingPolicy=[Name=XOR-2-1-1024k, Schema=[ECSchema=[Codec=xor, numDataUnits=2, numParityUnits=1]], CellSize=1048576, Id=4], State=DISABLED- Select the appropriate policy for the use case “
dfs ec -enablePolicy -policy XOR-2-1-1024k”.
x
[root@odp-krb1 keytabs]# hdfs ec -enablePolicy -policy XOR-2-1-1024kErasure coding policy XOR-2-1-1024k is enabled- Set policy
XOR-2-1-1024kon tmp directory “hdfs ec -setPolicy -path /tmp -policy XOR-2-1-1024k”.
[root@odp-krb1 keytabs]# hdfs ec -setPolicy -path /tmp -policy XOR-2-1-1024kSet XOR-2-1-1024k erasure coding policy on /tmpWarning: setting erasure coding policy on a non-empty directory will not automatically convert existing files to XOR-2-1-1024k erasure coding policy- Validate the set policy on the path.
hdfs ec -getPolicy -path /tmpXOR-2-1-1024k- Unset policy on the hdfs directory.
[root@odp-krb1 keytabs]# hdfs ec -unsetPolicy -path /tmpUnset erasure coding policy from /tmpWarning: unsetting erasure coding policy on a non-empty directory will not automatically convert existing files to replicated data.- Disable the set policy on HDFS.
[root@odp-krb1 keytabs]# hdfs ec -disablePolicy -policy XOR-2-1-1024kErasure coding policy XOR-2-1-1024k is disabledWas this page helpful?