Troubleshooting

Master Not Starting

# Check master logs

tail -f /var/log/celeborn/celeborn-master-*.log

Common causes:

• Port conflict — ensure ports 9097, 9098, and 9872 are available
• Java heap too small — increase CELEBORN_MASTER_MEMORY
• Ratis storage permission issue — check /var/lib/celeborn/ratis ownership

---

Worker Not Registering

# Check worker logs

tail -f /var/log/celeborn/celeborn-worker-*.log

Common causes:

• Cannot connect to master — verify network connectivity to port 9097
• Storage directory permission issues — ensure celeborn user has write access
• Disk full or unhealthy — check disk health and free space

---

Shuffle Failures

# Increase RPC and commit timeouts
celeborn.rpc.askTimeout=300s
celeborn.worker.commitFiles.timeout=300s

# Enable replication to prevent data loss on worker failure
spark.celeborn.client.push.replicate.enabled=true

---

Diagnostic REST API Commands

# Check master status and HA role
curl http://master:9098/api/v1/masters

# List all workers and their health status
curl http://master:9098/api/v1/workers

# List active shuffles
curl http://master:9098/api/v1/shuffles

# Health check
curl http://master:9098/ping

---

Log Locations