Configure Storage

Local Disk Storage

Bash
    
xxxxxxxxxx
 
# Multiple disks with disk type annotation celeborn.worker.storage.dirs=/mnt/nvme1:disktype=SSD,/mnt/nvme2:disktype=SSD,/mnt/hdd1:disktype=HDDceleborn.storage.availableTypes=SSD,HDD
Copy

HDFS Storage

The Real-World Use Case for HDFS in Celeborn

HDFS storage in Celeborn is designed for:

Disk exhaustion fallback - When local disk is full, new files go to HDFS
Not for primary tiered storage - The current architecture doesn't support MEMORY → HDFS eviction when local disk is available

Bash
    
xxxxxxxxxx
 
celeborn.storage.availableTypes=HDFS celeborn.storage.hdfs.dir=hdfs://namenode:8020/celeborn   # Kerberos authentication for HDFS celeborn.storage.hdfs.kerberos.principal=celeborn/_HOST@REALMceleborn.storage.hdfs.kerberos.keytab=/etc/security/keytabs/celeborn.keytab   # HDFS performance tuning (buffer must be >= 4 MB) celeborn.worker.flusher.hdfs.buffer.size=4mceleborn.worker.sortPartition.threads=64celeborn.worker.commitFiles.threads=128celeborn.rpc.askTimeout=240s
Copy

S3 / Object Storage

Bash
    
xxxxxxxxxx
 
celeborn.storage.availableTypes=S3celeborn.storage.s3.dir=s3a://bucket-name/celebornceleborn.storage.s3.endpoint.region=us-east-1 # S3 performance (larger buffer recommended)celeborn.worker.flusher.s3.buffer.size=6mceleborn.worker.flusher.s3.threads=8
Copy

For S3-compatible storage (Ceph, VAST, MinIO), configure the S3A endpoint in Hadoop core-site.xml:

Bash
    
xxxxxxxxxx
 
<property>   <name>fs.s3a.endpoint</name>   <value>http://your-s3-endpoint:port</value> </property> <property>   <name>fs.s3a.path.style.access</name>   <value>true</value> </property>
Copy

Tiered Storage

Configure eviction policies for hybrid SSD-to-HDFS tiering:

Bash
    
xxxxxxxxxx
 
celeborn.storage.availableTypes=SSD,HDFS celeborn.worker.storage.storagePolicy.createFilePolicy=SSD,HDFS celeborn.worker.storage.storagePolicy.evictPolicy=SSD|HDFS
Copy

Last updated on May 12, 2026

Was this page helpful?