Capacity Planning

Hardware Resource Ratio

Recommended balanced ratio per worker node for optimal Celeborn performance:

Resource	Ratio
vCPUs	2
Memory (GB)	5
Network (Gbps)	2
Disk I/O (KIOps)	1

Worker Sizing Formula

Bash
    
xxxxxxxxxx
 
Worker Count Calculation Formula: Worker Count = (Max Concurrent Shuffle Size × 2) / Node Usable Disk Space Example: Max concurrent shuffle = 1 TB, Node disk = 500 GB   → Workers needed = (1,000 GB × 2) / 500 GB = 4 workers
Copy

Memory Recommendations by Workload

Workload Profile	Worker Heap	Worker Off-Heap
Light (< 100 GB shuffle)	2 GB	4 GB
Medium (100 GB – 1 TB)	4 GB	8 GB
Heavy (> 1 TB)	8 GB	16 GB
HDFS / S3 backend	8 GB	32 GB

Storage Backend Recommendations

Storage Type	Performance	Notes
NVMe SSD	Best	Lowest latency — recommended for hot shuffle data
SAS SSD	Good	Good balance of cost and performance
HDD	Acceptable	Use multiple disks per worker for parallelism
HDFS	Moderate	Higher latency but infinite capacity — ideal for cold/large shuffles
S3 / Object Store	Lower	Highest latency, lowest cost — use for overflow or archival

###

Last updated on Apr 23, 2026

Was this page helpful?