Engine Integration

Apache Spark

Deploy Client JAR

Bash
    
xxxxxxxxxx
 
cp $CELEBORN_HOME/spark/celeborn-client-spark-3-shaded_2.12-0.6.2.jar $SPARK_HOME/jars/
Copy

Core Spark Configuration

Bash
    
xxxxxxxxxx
 
# Core settings spark.shuffle.manager=org.apache.spark.shuffle.celeborn.SparkShuffleManagerspark.celeborn.master.endpoints=master1:9097,master2:9097,master3:9097spark.shuffle.service.enabled=falsespark.serializer=org.apache.spark.serializer.KryoSerializer # Performance tuning spark.celeborn.client.push.replicate.enabled=truespark.celeborn.client.spark.shuffle.writer=hashspark.sql.adaptive.localShuffleReader.enabled=false # Dynamic allocation (Spark 3.5+) spark.dynamicAllocation.enabled=truespark.dynamicAllocation.shuffleTracking.enabled=falsespark.shuffle.sort.io.plugin.class=org.apache.spark.shuffle.celeborn.CelebornShuffleDataIO
Copy

Quick Test with spark-shell

Bash
    
 
/usr/odp/3.3.6.4-1/spark3/bin/spark-shell --conf spark.shuffle.manager=org.apache.spark.shuffle.celeborn.SparkShuffleManager --conf spark.shuffle.service.enabled=false --conf spark.sql.adaptive.localShuffleReader.enabled=false --conf spark.dynamicAllocation.shuffleTracking.enabled=false
Copy

Apache Flink

Deploy Client JAR.

Bash
    
xxxxxxxxxx
 
cp $CELEBORN_HOME/flink/celeborn-client-flink-1.19-shaded_2.12-0.6.2.jar $FLINK_HOME/lib/
Copy

Configure Flink.

Bash
    
xxxxxxxxxx
 
shuffle-service-factory.class=org.apache.celeborn.plugin.flink.RemoteShuffleServiceFactoryexecution.batch-shuffle-mode=ALL_EXCHANGES_BLOCKINGceleborn.master.endpoints=master1:9097,master2:9097,master3:9097 # Performance celeborn.client.shuffle.batchHandleReleasePartition.enabled=trueceleborn.client.push.maxReqsInFlight=128celeborn.data.io.numConnectionsPerPeer=16celeborn.data.io.threads=32
Copy

Apache Tez

Append following property in yarn-site xml under yarn.application.classpath and under Mapreduce mapreduce.application.classpath.

Bash
    
 
/usr/odp/3.3.6.4-1/celeborn/tez/celeborn-client-tez-shaded_2.12-0.6.2.3.3.6.4-1.jar:/usr/odp/3.3.6.4-1/celeborn/mr/celeborn-client-mr-shaded_2.12-0.6.2.3.3.6.4-1.jar
Copy

Bash
    
 
# In tez-site.xml tez.am.launch.cluster-default.cmd-opts=org.apache.tez.dag.app.CelebornDagAppMaster tez.celeborn.master.endpoints=master1:9097,master2:9097,master3:9097 tez.cluster.additional.classpath.prefix append  /usr/odp/3.3.6.3-101/celeborn/tez/celeborn-client-tez-shaded_2.12-0.6.2.3.3.6.3-101.jar
Copy

MapReduce

Need to add above tez configuration for classpath addition and then following additional properties in Mapreduce configuration.

Bash
    
xxxxxxxxxx
 
yarn.app.mapreduce.am.command-opts=org.apache.celeborn.mapreduce.v2.app.MRAppMasterWithCeleborn mapreduce.job.reduce.slowstart.completedmaps=1 mapreduce.celeborn.master.endpoints=master1:9097,master2:9097,master3:9097mapreduce.job.map.output.collector.class=org.apache.hadoop.mapred.CelebornMapOutputCollectormapreduce.job.reduce.shuffle.consumer.plugin.class=org.apache.hadoop.mapreduce.task.reduce.CelebornShuffleConsumeryarn.app.mapreduce.am.job.recovery.enable=false
Copy

Celeborn Master achieves HA using Apache Ratis (Raft consensus protocol). A minimum of 3 Master nodes is required. Odd numbers (3, 5, or 7) are required so the cluster can elect a leader by majority vote.

Bash
    
xxxxxxxxxx
 
HA Requirement Minimum: 3 Master nodes. Always use odd numbers (3, 5, 7). Ratis storage: /var/lib/celeborn/ratis — ensure this path has adequate I/O performance. Worker fault tolerance: Enable replication (spark.celeborn.client.push.replicate.enabled=true).
Copy

Master HA Configuration

Bash
    
xxxxxxxxxx
 
# Enable HA modeceleborn.master.ha.enabled=true # Node 1celeborn.master.ha.node.1.host=master1.example.comceleborn.master.ha.node.1.port=9097celeborn.master.ha.node.1.ratis.port=9872 # Node 2celeborn.master.ha.node.2.host=master2.example.comceleborn.master.ha.node.2.port=9097celeborn.master.ha.node.2.ratis.port=9872 # Node 3celeborn.master.ha.node.3.host=master3.example.comceleborn.master.ha.node.3.port=9097celeborn.master.ha.node.3.ratis.port=9872 # Ratis metadata storageceleborn.master.ha.ratis.raft.server.storage.dir=/var/lib/celeborn/ratis
Copy

Ambari Mpack HA Settings

Property (in celeborn-ha)	Value
celeborn.master.ha.enabled	true
celeborn.master.ha.node.mapping	master1:1,master2:2,master3:3
celeborn.master.ha.rpc.port	9097
celeborn.master.ha.ratis.port	9872

Worker Fault Tolerance

Enable data replication on the Spark client to ensure shuffle data survives a worker node failure:

Bash
    
xxxxxxxxxx
 
spark.celeborn.client.push.replicate.enabled=true
Copy

Last updated on Apr 23, 2026

Was this page helpful?