Using Multiple Spark Versions

Submitting Jobs to Specific Versions

Using spark-submit

Spark 3.5.5:

/usr/odp/current/spark3-client/bin/spark-submit \
 --class com.example.MyApp \
 --master yarn \
 --deploy-mode cluster \
 myapp.jar

Spark 3.3.3:

/usr/odp/current/spark3_3_3_3-client/bin/spark-submit \
 --class com.example.MyApp \
 --master yarn \
 --deploy-mode cluster \
 myapp.jar

Spark 3.5.1:

/usr/odp/current/spark3_3_5_1-client/bin/spark-submit \
 --class com.example.MyApp \
 --master yarn \
 --deploy-mode cluster \
 myapp.jar

Setting Environment Variables

Add to your application launcher script:

For Spark 3.5.5:

export SPARK_HOME=/usr/odp/current/spark3-client export SPARK_CONF_DIR=/etc/spark3/conf

For Spark 3.3.3:

export SPARK_HOME=/usr/odp/current/spark3_3_3_3-client export SPARK_CONF_DIR=/etc/spark3_3_3_3/conf

For Spark 3.5.1:

export SPARK_HOME=/usr/odp/current/spark3_3_5_1-client export SPARK_CONF_DIR=/etc/spark3_3_5_1/conf