Troubleshooting

Master Not Starting

Bash
    
xxxxxxxxxx
 
# Check master logs tail -f /var/log/celeborn/celeborn-master-*.log
Copy

Common causes:

Port conflict — ensure ports 9097, 9098, and 9872 are available
Java heap too small — increase CELEBORN_MASTER_MEMORY
Ratis storage permission issue — check /var/lib/celeborn/ratis ownership

Worker Not Registering

Bash
    
xxxxxxxxxx
 
# Check worker logs tail -f /var/log/celeborn/celeborn-worker-*.log
Copy

Common causes:

Cannot connect to master — verify network connectivity to port 9097
Storage directory permission issues — ensure celeborn user has write access
Disk full or unhealthy — check disk health and free space

Shuffle Failures

Bash
    
xxxxxxxxxx
 
# Increase RPC and commit timeoutsceleborn.rpc.askTimeout=300sceleborn.worker.commitFiles.timeout=300s # Enable replication to prevent data loss on worker failurespark.celeborn.client.push.replicate.enabled=true
Copy

Diagnostic REST API Commands

Bash
    
xxxxxxxxxx
 
# Check master status and HA rolecurl http://master:9098/api/v1/masters # List all workers and their health statuscurl http://master:9098/api/v1/workers # List active shufflescurl http://master:9098/api/v1/shuffles # Health checkcurl http://master:9098/ping
Copy

Log Locations

Component	Log Path
Celeborn Master	/var/log/celeborn/celeborn-master-*.log
Celeborn Worker	/var/log/celeborn/celeborn-worker-*.log
Ambari Server	/var/log/ambari-server/ambari-server.log

Last updated on Apr 23, 2026

Was this page helpful?