Prerequisites for Celeborn

Master Nodes

1

3 (for HA)

Worker Nodes

1

3 or more

CPU per Worker

4 cores

8–16 cores

Memory per Worker

4 GB heap + 4 GB off-heap

8 GB heap + 16 GB off-heap

Disk per Worker

100 GB SSD

500 GB+ SSD / NVMe

Network

1 Gbps

10 Gbps

9097

Master RPC

TCP

Client / Worker to Master communication

9098

Master HTTP

TCP

Web UI and REST API

9872

Ratis

TCP

Master-to-Master Raft consensus

9094

Worker RPC

TCP

Client to Worker data transfer

9096

Worker HTTP

TCP

Worker Web UI and Prometheus metrics

19097

Master Internal

TCP

Internal port when auth is enabled