Troubleshooting

Master Not Starting

# Check master logs tail -f /var/log/celeborn/celeborn-master-*.log

Common causes:

Port conflict — ensure ports 9097, 9098, and 9872 are available
Java heap too small — increase CELEBORN_MASTER_MEMORY
Ratis storage permission issue — check /var/lib/celeborn/ratis ownership

Worker Not Registering

# Check worker logs tail -f /var/log/celeborn/celeborn-worker-*.log

Common causes:

Cannot connect to master — verify network connectivity to port 9097
Storage directory permission issues — ensure celeborn user has write access
Disk full or unhealthy — check disk health and free space

Shuffle Failures

# Increase RPC and commit timeouts celeborn.rpc.askTimeout=300s celeborn.worker.commitFiles.timeout=300s # Enable replication to prevent data loss on worker failure spark.celeborn.client.push.replicate.enabled=true

Diagnostic REST API Commands

# Check master status and HA role curl http://master:9098/api/v1/masters # List all workers and their health status curl http://master:9098/api/v1/workers # List active shuffles curl http://master:9098/api/v1/shuffles # Health check curl http://master:9098/ping

Log Locations

Component	Log Path
Celeborn Master	/var/log/celeborn/celeborn-master-*.log
Celeborn Worker	/var/log/celeborn/celeborn-worker-*.log
Ambari Server	/var/log/ambari-server/ambari-server.log

Last updated on Apr 23, 2026

Was this page helpful?