Prerequisites for Celeborn

System Requirements

ComponentMinimumRecommended
Master Nodes13 (for HA)
Worker Nodes13 or more
CPU per Worker4 cores8–16 cores
Memory per Worker4 GB heap + 4 GB off-heap8 GB heap + 16 GB off-heap
Disk per Worker100 GB SSD500 GB+ SSD / NVMe
Network1 Gbps10 Gbps

Software Requirements

  • ODP Stack 3.3.6.x or higher
  • Ambari 2.7.x or higher
  • Java 8 or Java 11
  • Hadoop 3.x (required for HDFS storage backend)

Network Ports

PortServiceProtocolDescription
9097Master RPCTCPClient / Worker to Master communication
9098Master HTTPTCPWeb UI and REST API
9872RatisTCPMaster-to-Master Raft consensus
9094Worker RPCTCPClient to Worker data transfer
9096Worker HTTPTCPWorker Web UI and Prometheus metrics
19097Master InternalTCPInternal port when auth is enabled
VariableType to search · ESC to discard
GlossaryType to search · ESC to discard
InsertType to search · ESC to discard
No matches
  Last updated