Configure Storage

Local Disk Storage

# Multiple disks with disk type annotation

celeborn.worker.storage.dirs=/mnt/nvme1:disktype=SSD,/mnt/nvme2:disktype=SSD,/mnt/hdd1:disktype=HDD
celeborn.storage.availableTypes=SSD,HDD

---

HDFS Storage

The Real-World Use Case for HDFS in Celeborn

HDFS storage in Celeborn is designed for:

• Disk exhaustion fallback - When local disk is full, new files go to HDFS
• Not for primary tiered storage - The current architecture doesn't support MEMORY → HDFS eviction when local disk is available

celeborn.storage.availableTypes=HDFS

celeborn.storage.hdfs.dir=hdfs://namenode:8020/celeborn

# Kerberos authentication for HDFS

celeborn.storage.hdfs.kerberos.principal=celeborn/_HOST@REALM
celeborn.storage.hdfs.kerberos.keytab=/etc/security/keytabs/celeborn.keytab

# HDFS performance tuning (buffer must be >= 4 MB)

celeborn.worker.flusher.hdfs.buffer.size=4m
celeborn.worker.sortPartition.threads=64
celeborn.worker.commitFiles.threads=128
celeborn.rpc.askTimeout=240s

S3 / Object Storage

celeborn.storage.availableTypes=S3
celeborn.storage.s3.dir=s3a://bucket-name/celeborn
celeborn.storage.s3.endpoint.region=us-east-1

# S3 performance (larger buffer recommended)
celeborn.worker.flusher.s3.buffer.size=6m
celeborn.worker.flusher.s3.threads=8

For S3-compatible storage (Ceph, VAST, MinIO), configure the S3A endpoint in Hadoop core-site.xml:

<property>

 <name>fs.s3a.endpoint</name>

 <value>http://your-s3-endpoint:port</value>

</property>

<property>

 <name>fs.s3a.path.style.access</name>

 <value>true</value>

</property>

---

Tiered Storage

Configure eviction policies for hybrid SSD-to-HDFS tiering:

celeborn.storage.availableTypes=SSD,HDFS

celeborn.worker.storage.storagePolicy.createFilePolicy=SSD,HDFS

celeborn.worker.storage.storagePolicy.evictPolicy=SSD|HDFS