Prerequisites for Celeborn

System Requirements

ComponentMinimumRecommended
Master Nodes13 (for HA)
Worker Nodes13 or more
CPU per Worker4 cores8–16 cores
Memory per Worker4 GB heap + 4 GB off-heap8 GB heap + 16 GB off-heap
Disk per Worker100 GB SSD500 GB+ SSD / NVMe
Network1 Gbps10 Gbps

Software Requirements

  • ODP Stack 3.3.6.x or higher
  • Ambari 2.7.x or higher
  • Java 8 or Java 11
  • Hadoop 3.x (required for HDFS storage backend)

Network Ports

PortServiceProtocolDescription
9097Master RPCTCPClient / Worker to Master communication
9098Master HTTPTCPWeb UI and REST API
9872RatisTCPMaster-to-Master Raft consensus
9094Worker RPCTCPClient to Worker data transfer
9096Worker HTTPTCPWorker Web UI and Prometheus metrics
19097Master InternalTCPInternal port when auth is enabled
Type to search, ESC to discard
Type to search, ESC to discard
Type to search, ESC to discard
  Last updated