Documentation
ODP 3.3.6.4-1
Release Notes
What is ODP
Installation
Component User guide and Installation Instructions
Upgrade Instructions
Downgrade Instructions
Reference Guide
Security Guide
Troubleshooting Guide
Uninstall ODP
Title
Message
Create new category
What is the title of your new category?
Edit page index title
What is the title of the page index?
Edit category
What is the new title of your category?
Edit link
What is the new title and URL of your link?
Configuration Reference
Summarize Page
Copy Markdown
Open in ChatGPT
Open in Claude
Connect to Cursor
Connect to VS Code
Master Configuration
Bash
xxxxxxxxxx# Identity celeborn.master.host=<hostname>celeborn.master.port=9097celeborn.master.http.port=9098 # Slot assignment (roundrobin or loadaware)celeborn.master.slot.assign.policy=loadaware # Worker health managementceleborn.master.heartbeat.timeout=120sceleborn.master.worker.timeout=600sWorker Configuration
Bash
xxxxxxxxxx# Storage — multiple disks with disk type annotation celeborn.worker.storage.dirs=/mnt/nvme1:disktype=SSD,/mnt/nvme2:disktype=SSD,/mnt/hdd1:disktype=HDDceleborn.storage.availableTypes=SSD,HDD # Memory (set in celeborn-env.sh) CELEBORN_WORKER_MEMORY=2gCELEBORN_WORKER_OFFHEAP_MEMORY=4g # Flush buffer sizes celeborn.worker.flusher.buffer.size=256kceleborn.worker.flusher.threads=4 # Threading (increase for HDFS/S3 workloads) celeborn.worker.sortPartition.threads=64celeborn.worker.commitFiles.threads=128celeborn.worker.commitFiles.timeout=240sClient Configuration (Spark)
Bash
xxxxxxxxxx# Enable Celeborn shuffle managerspark.shuffle.manager=org.apache.spark.shuffle.celeborn.SparkShuffleManagerspark.celeborn.master.endpoints=master1:9097,master2:9097,master3:9097 # Disable YARN ESS (required when using Celeborn)spark.shuffle.service.enabled=false # Serialization (required for Celeborn)spark.serializer=org.apache.spark.serializer.KryoSerializer # Replication (recommended for production)spark.celeborn.client.push.replicate.enabled=true # Writer mode: hash (moderate partitions) or sort (high partitions)spark.celeborn.client.spark.shuffle.writer=hash # Dynamic allocation (Spark 3.5+)spark.dynamicAllocation.enabled=truespark.dynamicAllocation.shuffleTracking.enabled=false spark.shuffle.sort.io.plugin.class=org.apache.spark.shuffle.celeborn.CelebornShuffleDataIOType to search, ESC to discard
Type to search, ESC to discard
Type to search, ESC to discard
Last updated on Apr 23, 2026
Was this page helpful?
Next to read:
Engine Integrationnull
Discard Changes
Do you want to discard your current changes and overwrite with the template?
Archive Synced Block
Message
Create new Template
What is this template's title?
Delete Template
Message