Component Diagram
Celeborn consists of three primary components: Masters (cluster coordinator), Workers (shuffle data storage), and Clients (embedded in compute engines).
Component Details
Master
Role: Cluster coordinator and metadata manager
Manages worker registration and health monitoring
Allocates shuffle slots to applications
Coordinates shuffle lifecycle and resource cleanup
Achieves HA via Raft consensus — 3, 5, or 7 nodes recommended
Port | Protocol | Purpose |
|---|---|---|
9097 | TCP | Master RPC — Client and Worker communication |
9098 | TCP | Master HTTP — Web UI and REST API |
9872 | TCP | Ratis — Master-to-Master Raft consensus |
Worker
Role: Shuffle data storage and serving
Receives and buffers shuffle data from mapper executors
Replicates data to peer workers for fault tolerance
Serves shuffle data chunks to reducer executors
Supports local disk, HDFS, S3, and tiered storage backends
Port | Protocol | Purpose |
|---|---|---|
9094 | TCP | Worker RPC — Client to Worker communication |
9096 | TCP | Worker HTTP — Worker Web UI and metrics |
Client
The client is embedded in compute engines and consists of two sub-components.
Component | Location | Responsibility |
|---|---|---|
LifecycleManager | Driver / JobMaste | Control plane — manages shuffle metadata and slot allocation |
ShuffleClient | Executor / TaskManager | Data plane — pushes and fetches shu |
Shuffle Lifecycle
Every shuffle operation follows a structured seven-step lifecycle managed between the Client, Master, and Workers:
Step | Direction | Description |
|---|---|---|
| Client → Master | Request slot allocation for the shuffle |
| Client → Workers | Reserve resources on selected workers |
| Executors → Workers | Mapper executors stream shuffle data to workers |
| Client → Workers | Flush buffers, finalize and commit partitions |
| Client → Workers | Prepare workers to serve data for reducers |
| Executors → Workers | Reducer executors fetch data chunks by partition |
| Client → Master | Release all resources after job completion |