Configure Storage

Local Disk Storage

# Multiple disks with disk type annotation celeborn.worker.storage.dirs=/mnt/nvme1:disktype=SSD,/mnt/nvme2:disktype=SSD,/mnt/hdd1:disktype=HDD celeborn.storage.availableTypes=SSD,HDD

HDFS Storage

The Real-World Use Case for HDFS in Celeborn

HDFS storage in Celeborn is designed for:

Disk exhaustion fallback - When local disk is full, new files go to HDFS
Not for primary tiered storage - The current architecture doesn't support MEMORY → HDFS eviction when local disk is available

celeborn.storage.availableTypes=HDFS celeborn.storage.hdfs.dir=hdfs://namenode:8020/celeborn # Kerberos authentication for HDFS celeborn.storage.hdfs.kerberos.principal=celeborn/_HOST@REALM celeborn.storage.hdfs.kerberos.keytab=/etc/security/keytabs/celeborn.keytab # HDFS performance tuning (buffer must be >= 4 MB) celeborn.worker.flusher.hdfs.buffer.size=4m celeborn.worker.sortPartition.threads=64 celeborn.worker.commitFiles.threads=128 celeborn.rpc.askTimeout=240s

S3 / Object Storage

celeborn.storage.availableTypes=S3 celeborn.storage.s3.dir=s3a://bucket-name/celeborn celeborn.storage.s3.endpoint.region=us-east-1 # S3 performance (larger buffer recommended) celeborn.worker.flusher.s3.buffer.size=6m celeborn.worker.flusher.s3.threads=8

For S3-compatible storage (Ceph, VAST, MinIO), configure the S3A endpoint in Hadoop core-site.xml:

<property> <name>fs.s3a.endpoint</name> <value>http://your-s3-endpoint:port</value> </property> <property> <name>fs.s3a.path.style.access</name> <value>true</value> </property>

Tiered Storage

Configure eviction policies for hybrid SSD-to-HDFS tiering:

celeborn.storage.availableTypes=SSD,HDFS celeborn.worker.storage.storagePolicy.createFilePolicy=SSD,HDFS celeborn.worker.storage.storagePolicy.evictPolicy=SSD|HDFS

Last updated on May 12, 2026

Was this page helpful?