Run Spark with Velox

You can use Velox with Spark 3.5.5 and Spark 4.1.1 to accelerate Spark SQL workloads. Gluten provides the Spark plugin that offloads supported SQL operations to the Velox native execution engine.

To enable Gluten and Velox, you must configure:

spark.plugins=org.apache.gluten.GlutenPlugin

Without this configuration, Velox isn't enabled.

Prerequisites

Before you submit an application:

  1. Set SPARK_HOME to the client for the Spark version that you want to use.

    For Spark 3.5.5:

    export SPARK_HOME=/usr/odp/current/spark3-client

    For Spark 4.1.1:

    export SPARK_HOME=/usr/odp/current/spark4-client

  2. Use Parquet or ORC data for better performance.

  3. Enable off-heap memory for Velox:

    spark.memory.offHeap.enabled=true

  4. Configure sufficient off-heap memory for the YARN container.

    The examples on this page use 2g. For larger workloads, increase the value as required.

Run Spark 3.5.5 with Velox

Gluten JAR

On ODP, the Gluten bundle is typically available in the Spark 3 jars directory:

/usr/odp/current/spark3-client/jars/gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.4.0.jar

Important: If the Gluten JAR is already in the Spark 3 jars directory, don't specify the same JAR by using --jars. Loading the JAR more than once can cause a duplicate VeloxBackend error.

Submit a Spark application

Set SPARK_HOME:

export SPARK_HOME=/usr/odp/current/spark3-client

On JDK 11 environments, configure the required Java options:

JAVAOPTS='-Dio.netty.tryReflectionSetAccessible=true --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.misc=ALL-UNNAMED'

Submit the application:

$SPARK_HOME/bin/spark-submit \ --master yarn \ --deploy-mode cluster \ --name my-app-spark355-velox \ --conf spark.plugins=org.apache.gluten.GlutenPlugin \ --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \ --conf spark.memory.offHeap.enabled=true \ --conf spark.memory.offHeap.size=2g \ --conf spark.gluten.enabled=true \ --conf spark.gluten.sql.columnar.backend.lib=velox \ --conf spark.sql.autoBroadcastJoinThreshold=-1 \ --conf spark.driver.extraJavaOptions="$JAVAOPTS" \ --conf spark.executor.extraJavaOptions="$JAVAOPTS" \ --class com.example.MyApp \ /path/to/my-app.jar

Run Spark SQL

export SPARK_HOME=/usr/odp/current/spark3-client JAVAOPTS='-Dio.netty.tryReflectionSetAccessible=true --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.misc=ALL-UNNAMED' $SPARK_HOME/bin/spark-sql --master yarn \ --name my-sql-spark355-velox \ --conf spark.plugins=org.apache.gluten.GlutenPlugin \ --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \ --conf spark.memory.offHeap.enabled=true \ --conf spark.memory.offHeap.size=2g \ --conf spark.gluten.enabled=true \ --conf spark.gluten.sql.columnar.backend.lib=velox \ --conf spark.sql.autoBroadcastJoinThreshold=-1 \ --conf spark.driver.extraJavaOptions="$JAVAOPTS" \ --conf spark.executor.extraJavaOptions="$JAVAOPTS" \ -e "SELECT 1"

Disable broadcast joins

On some JDK 11 environments, Gluten columnar broadcast operations can fail with the following error:

DirectByteBuffer.<init>(long, int) not available

To avoid this issue, the preceding examples disable broadcast joins:

spark.sql.autoBroadcastJoinThreshold=-1

You can enable broadcast joins if your JDK and Gluten build support them.

Run Spark 4.1.1 with Velox

Spark 4.1.1 has the following requirements that differ from Spark 3.5.5:

  • Spark 4.1.1 requires JDK 17 for the YARN ApplicationMaster and executors.

  • The Gluten JAR might not be in the Spark 4 jars directory. In this case, specify it by using --jars.

Configure JDK 17

Set the Spark 4 client:

export SPARK_HOME=/usr/odp/current/spark4-client

Set the JDK 17 installation directory:

export JDK17=/usr/lib/jvm/java-17-openjdk

If multiple JDK 17 installations are available, specify the complete installation path.

For example:

export JDK17=/usr/lib/jvm/java-17-openjdk-17.0.20.1.1-1.1.el8_10.x86_64

Locate the Gluten JAR

The Gluten JAR is typically available at a path similar to:

/usr/odp/3.3.6.5-<build>/spark4/gluten/gluten-velox-bundle-spark4.1_2.13-linux_amd64-1.7.0.3.3.6.5.jar

The build number can vary depending on your ODP installation.

You can locate the JAR by running:

export GLUTEN=$(ls /usr/odp/*/spark4/gluten/gluten-velox-bundle-spark4.1*.jar 2>/dev/null | head -1) echo "Using GLUTEN=$GLUTEN"

Submit a Spark application

$SPARK_HOME/bin/spark-submit \ --master yarn \ --deploy-mode cluster \ --name my-app-spark411-velox \ --jars "$GLUTEN" \ --conf spark.plugins=org.apache.gluten.GlutenPlugin \ --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \ --conf spark.memory.offHeap.enabled=true \ --conf spark.memory.offHeap.size=2g \ --conf spark.gluten.enabled=true \ --conf spark.gluten.sql.columnar.backend.lib=velox \ --conf spark.yarn.appMasterEnv.JAVA_HOME="$JDK17" \ --conf spark.executorEnv.JAVA_HOME="$JDK17" \ --class com.example.MyApp \ /path/to/my-app.jar

Run Spark SQL

export SPARK_HOME=/usr/odp/current/spark4-client export JDK17=/usr/lib/jvm/java-17-openjdk export GLUTEN=$(ls /usr/odp/*/spark4/gluten/gluten-velox-bundle-spark4.1*.jar 2>/dev/null | head -1) $SPARK_HOME/bin/spark-sql --master yarn \ --name my-sql-spark411-velox \ --jars "$GLUTEN" \ --conf spark.plugins=org.apache.gluten.GlutenPlugin \ --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \ --conf spark.memory.offHeap.enabled=true \ --conf spark.memory.offHeap.size=2g \ --conf spark.gluten.enabled=true \ --conf spark.gluten.sql.columnar.backend.lib=velox \ --conf spark.yarn.appMasterEnv.JAVA_HOME="$JDK17" \ --conf spark.executorEnv.JAVA_HOME="$JDK17" \ -e "SELECT 1"

Note: When you use spark-sql with Spark 4.1.1, don't specify both -e and -f in the same command. Use either -e or -f.

Velox configuration

The following configurations apply to both Spark versions:

Configuration

Value

Description

spark.plugins

org.apache.gluten.GlutenPlugin

Loads the Gluten plugin.

spark.gluten.enabled

true

Enables Gluten.

spark.gluten.sql.columnar.backend.lib

velox

Uses Velox as the columnar backend.

spark.memory.offHeap.enabled

true

Enables off-heap memory for native processing.

spark.memory.offHeap.size

2g–12g

Specifies the amount of off-heap memory available to the native buffer pool.

spark.shuffle.manager

org.apache.spark.shuffle.sort.ColumnarShuffleManager

Enables the columnar shuffle manager.

You can also configure the following optional properties:

spark.sql.adaptive.enabled=true spark.gluten.sql.columnar.backend.velox.spillEnabled=true spark.gluten.sql.columnar.backend.velox.spillPath=/tmp/velox-spill

Verify that Velox is enabled

After you submit an application, verify that Gluten and Velox loaded successfully.

  1. Open the Spark UI.

  2. Go to Environment.

  3. Verify that the following configuration is present:

    spark.plugins=org.apache.gluten.GlutenPlugin

  4. Run df.explain(true) or EXPLAIN EXTENDED.

  5. Check the execution plan for Gluten operators, such as:

    GlutenScan GlutenFilter GlutenProject

  6. Check the driver logs for messages that indicate that Gluten and Velox loaded successfully.

If the execution plan contains only standard Spark operators and doesn't contain Gluten operators, verify the Gluten configuration and JAR location.

Troubleshoot Velox

Issue

Resolution

Gluten plugin isn't found

Verify SPARK_HOME. For Spark 4.1.1, also verify that the Gluten JAR is specified with --jars.

duplicate VeloxBackend error

Don't specify the Gluten bundle with --jars if it's already in the Spark 3 jars directory.

Module jdk.incubator.vector not found with Spark 4.1.1

Configure JDK 17 for the ApplicationMaster and executors.

Out-of-memory or memory allocation errors

Increase spark.memory.offHeap.size or enable Velox spilling.

Incorrect results or unsupported query

Set spark.gluten.enabled=false and run the application without Gluten.

History Server errors after adding Gluten to the Spark 4 jars directory

Keep the Gluten JAR outside the Spark 4 jars directory and specify it with --jars for individual applications.

Disable Velox

To run an application without Velox, remove the spark.plugins and Gluten-specific configurations.

Alternatively, disable Gluten for the application:

--conf spark.gluten.enabled=false

The application then runs using the standard Spark execution engine.

  Last updated on Aug 31, 2026
Next to read:
Access Hive Tables with Spark
null
On This Page
Run Spark with VeloxPrerequisitesRun Spark 3.5.5 with VeloxRun Spark 4.1.1 with VeloxVelox configurationVerify that Velox is enabledTroubleshoot VeloxDisable VeloxGluten JARSubmit a Spark applicationRun Spark SQLDisable broadcast joinsConfigure JDK 17Locate the Gluten JARSubmit a Spark applicationRun Spark SQL