Gluten with Velox

Apache Gluten with Velox backend is a native execution plugin for Apache Spark that accelerates SQL query performance without requiring any code changes to your existing Spark applications.

Key Benefits

• Zero Code Changes – Works with existing Spark SQL, DataFrame, and Dataset APIs
• Significant Performance Gains – 2–5× faster on analytical workloads
• Drop-in Replacement – Simply add JAR and configuration
• Transparent Acceleration – Automatically offloads supported operations to native Velox engine
• Production Ready – Battle-tested on TPC-DS, TPC-H benchmarks

What is Gluten?

Apache Gluten is a Spark plugin that offloads SQL query execution from JVM to native C++ execution engines. It acts as a “glue” layer between Apache Spark and vectorized execution engines like Velox.

Architecture Overview

┌─────────────────────────────────────────────────┐
│         Your Spark Application                  │
│     (SQL, DataFrame, Dataset APIs)              │
│              NO CHANGES NEEDED                  │
└─────────────────────────────────────────────────┘
                      ↓
┌─────────────────────────────────────────────────┐
│         Apache Spark SQL Engine                 │
│         (Query Planning & Optimization)         │
└─────────────────────────────────────────────────┘
                      ↓
┌─────────────────────────────────────────────────┐
│            Gluten Plugin Layer                  │
│   (Translates Spark Plans to Native Plans)      │
└─────────────────────────────────────────────────┘
                      ↓
┌─────────────────────────────────────────────────┐
│         Velox Execution Engine (C++)            │
│    (Vectorized Processing, SIMD, Zero-Copy)     │
└─────────────────────────────────────────────────┘

Why Velox?

Velox is Meta's unified execution engine that powers:

• Meta's data warehouse (Presto)
• Stream processing systems
• Machine learning infrastructure

Key Features

• Vectorized execution – processes data in batches
• SIMD instructions – hardware-accelerated operations
• Advanced codegen – runtime code generation
• Memory efficiency – Arrow-based zero-copy design

How It Works

Transparent Query Acceleration

When you enable Gluten, Spark automatically:

• Analyzes your SQL query plan
• Identifies operations that can be accelerated (scans, filters, joins, aggregations)
• Offloads those operations to the Velox native engine
• Falls back to Spark for unsupported operations
• Returns results – your application receives the same output, just faster!

Supported Operations

✅ Fully Supported

• Table scans (Parquet, ORC, CSV)
• Filters and predicates
• Projections
• Hash joins (inner, left, right, full outer, semi, anti)
• Hash aggregations (sum, avg, min, max, count)
• Sort operations
• Window functions
• String operations
• Date/time operations
• Math functions

⚠️ Partial Support

• Some complex UDFs
• Certain window function combinations
• Specialized data types

❌ Not Supported

• Custom data sources
• Complex nested UDFs
• Some exotic data types

Performance Expectations

Based on TPC-DS and production workloads:

Real-world Examples

• TPC-DS Query 72: 45s → 12s (3.75× speedup)
• TPC-DS Query 95: 120s → 35s (3.4× speedup)

---

Getting Started

Prerequisites

Step 1: Obtain Gluten JAR

Contact your Acceldata account manager or download from:

gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.4.0.jar

Place JAR in:

/usr/odp/current/spark3-client/jars/

Step 2: Enable Gluten (No Code Changes!)

Option A: Via spark-submit

spark-submit --jars /usr/odp/current/spark3-client/jars/gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.4.0.jar --conf spark.plugins=org.apache.gluten.GlutenPlugin --conf spark.memory.offHeap.enabled=true --conf spark.memory.offHeap.size=8g --conf spark.gluten.sql.enabled=true --conf spark.gluten.sql.columnar.backend.lib=velox your-application.jar

Option B: Via spark-defaults.conf

Add to $SPARK_HOME/conf/spark-defaults.conf:

spark.jars                                        /usr/odp/current/spark3-client/jars/gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.4.0.jar
spark.plugins                                     org.apache.gluten.GlutenPlugin
spark.memory.offHeap.enabled                      true
spark.memory.offHeap.size                         8g
spark.gluten.sql.enabled                          true
spark.gluten.sql.columnar.backend.lib             velox

Option C: Programmatic Configuration (Scala/Java)

val spark = SparkSession.builder()
 .appName("MyApp")
 .config("spark.plugins", "org.apache.gluten.GlutenPlugin")
 .config("spark.memory.offHeap.enabled", "true")
 .config("spark.memory.offHeap.size", "8g")
 .config("spark.gluten.sql.enabled", "true")
 .config("spark.gluten.sql.columnar.backend.lib", "velox")
 .getOrCreate()
// Your existing code runs unchanged!
spark.sql("SELECT * FROM large_table WHERE value > 100").show()

Option D: PySpark

from pyspark.sql import SparkSession
spark = SparkSession.builder .appName("MyApp") .config("spark.plugins", "org.apache.gluten.GlutenPlugin") .config("spark.memory.offHeap.enabled", "true") .config("spark.memory.offHeap.size", "8g") .config("spark.gluten.sql.enabled", "true") .config("spark.gluten.sql.columnar.backend.lib", "velox") .getOrCreate()
# Your existing code runs unchanged!
df = spark.sql("SELECT * FROM large_table WHERE value > 100")
df.show()

Step 3: Run Your Application

That's it! Your application runs exactly as before, but faster.

---

Configuration Guide

Minimal Configuration (Quick Start)

For JDK 11

spark-submit --jars /usr/odp/current/spark3-client/jars/gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.4.0.jar --conf spark.plugins=org.apache.gluten.GlutenPlugin --conf spark.memory.offHeap.enabled=true --conf spark.memory.offHeap.size=8g --conf spark.gluten.sql.enabled=true --conf spark.gluten.sql.columnar.backend.lib=velox --conf spark.driver.extraJavaOptions="--add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED" --conf spark.executor.extraJavaOptions="--add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED" your-application.jar

For JDK 8

spark-submit --jars /usr/odp/current/spark3-client/jars/gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.4.0.jar --conf spark.plugins=org.apache.gluten.GlutenPlugin --conf spark.memory.offHeap.enabled=true --conf spark.memory.offHeap.size=8g --conf spark.gluten.sql.enabled=true --conf spark.gluten.sql.columnar.backend.lib=velox your-application.jar

Recommended Production Configuration

spark-submit --master yarn --deploy-mode cluster --driver-memory 4g --executor-memory 16g --executor-cores 4 --num-executors 10 --jars /usr/odp/current/spark3-client/jars/gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.4.0.jar # Core Gluten Settings
 --conf spark.plugins=org.apache.gluten.GlutenPlugin --conf spark.gluten.sql.enabled=true --conf spark.gluten.sql.columnar.backend.lib=velox # Memory Configuration
 --conf spark.memory.offHeap.enabled=true --conf spark.memory.offHeap.size=12g --conf spark.gluten.memory.fraction=0.7 # Shuffle Configuration
 --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager --conf spark.gluten.sql.columnar.shuffle.codec=lz4 # Adaptive Query Execution
 --conf spark.sql.adaptive.enabled=true --conf spark.sql.adaptive.coalescePartitions.enabled=true --conf spark.sql.adaptive.skewJoin.enabled=true # Velox-Specific Settings
 --conf spark.gluten.sql.columnar.backend.velox.spillEnabled=true --conf spark.gluten.sql.columnar.backend.velox.spillPath=/tmp/velox-spill # Java Options (JDK 11 only)
 --conf spark.driver.extraJavaOptions="--add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED" --conf spark.executor.extraJavaOptions="--add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED" your-application.jar

TPC-DS Benchmark Configuration (Tested)

This is the exact configuration used by Acceldata for TPC-DS benchmarking.

For JDK 11

spark-sql --driver-memory 3g --executor-memory 12g --executor-cores 3 --num-executors 9 --jars /usr/odp/current/spark3-client/jars/gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.4.0.jar --conf spark.plugins=org.apache.gluten.GlutenPlugin --conf spark.sql.optimizer.runtime.bloomFilter.applicationSideScanSizeThreshold=128MB --conf spark.memory.offHeap.enabled=true --conf spark.memory.offHeap.size=8g --conf spark.gluten.sql.enabled=true --conf spark.gluten.sql.columnar.backend.lib=velox --conf spark.gluten.sql.columnar.backend.velox=true --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager --conf spark.gluten.sql.columnar.shuffle.codec=lz4 --conf spark.driver.extraJavaOptions="--illegal-access=permit -Dio.netty.tryReflectionSetAccessible=true --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED" --conf spark.executor.extraJavaOptions="--illegal-access=permit -Dio.netty.tryReflectionSetAccessible=true --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED" --conf spark.sql.adaptive.enabled=true --conf spark.sql.adaptive.coalescePartitions.enabled=true --conf spark.sql.adaptive.skewJoin.enabled=true --conf spark.sql.adaptive.localShuffleReader.enabled=true --conf spark.gluten.sql.columnar.backend.velox.memCacheSize=4g --conf spark.sql.inMemoryColumnarStorage.compressed=true --conf spark.sql.inMemoryColumnarStorage.batchSize=50000 --conf spark.gluten.sql.columnar.forceShuffledHashJoin=false --conf spark.gluten.memory.fraction=0.7 --conf spark.gluten.sql.columnar.backend.velox.maxBatchSize=32768 --conf spark.gluten.sql.columnar.backend.velox.bloomFilterEnabled=true --conf spark.gluten.sql.columnar.preferColumnar=true --conf spark.gluten.sql.columnar.backend.velox.aggregationPreferredSize=1048576 --conf spark.sql.adaptive.advisoryPartitionSizeInBytes=128MB --conf spark.sql.adaptive.maxShuffledHashJoinLocalMapThreshold=1GB --conf spark.gluten.sql.columnar.backend.velox.spillEnabled=true --conf spark.gluten.sql.columnar.backend.velox.spillPath=/tmp/velox-spill --conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.sql.execution.arrow.pyspark.enabled=true --database tpcds_parquet -f file:///home/hive/hive-testbench/spark-queries-tpcds/query42.sql

For JDK 8

spark-sql --driver-memory 3g --executor-memory 12g --executor-cores 3 --num-executors 9 --jars /usr/odp/current/spark3-client/jars/gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.4.0.jar --conf spark.plugins=org.apache.gluten.GlutenPlugin --conf spark.sql.optimizer.runtime.bloomFilter.applicationSideScanSizeThreshold=128MB --conf spark.memory.offHeap.enabled=true --conf spark.memory.offHeap.size=8g --conf spark.gluten.sql.enabled=true --conf spark.gluten.sql.columnar.backend.lib=velox --conf spark.gluten.sql.columnar.backend.velox=true --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager --conf spark.gluten.sql.columnar.shuffle.codec=lz4 --conf spark.sql.adaptive.enabled=true --conf spark.sql.adaptive.coalescePartitions.enabled=true --conf spark.sql.adaptive.skewJoin.enabled=true --conf spark.sql.adaptive.localShuffleReader.enabled=true --conf spark.gluten.sql.columnar.backend.velox.memCacheSize=4g --conf spark.sql.inMemoryColumnarStorage.compressed=true --conf spark.sql.inMemoryColumnarStorage.batchSize=50000 --conf spark.gluten.sql.columnar.forceShuffledHashJoin=false --conf spark.gluten.memory.fraction=0.7 --conf spark.gluten.sql.columnar.backend.velox.maxBatchSize=32768 --conf spark.gluten.sql.columnar.backend.velox.bloomFilterEnabled=true --conf spark.gluten.sql.columnar.preferColumnar=true --conf spark.gluten.sql.columnar.backend.velox.aggregationPreferredSize=1048576 --conf spark.sql.adaptive.advisoryPartitionSizeInBytes=128MB --conf spark.sql.adaptive.maxShuffledHashJoinLocalMapThreshold=1GB --conf spark.gluten.sql.columnar.backend.velox.spillEnabled=true --conf spark.gluten.sql.columnar.backend.velox.spillPath=/tmp/velox-spill --conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.sql.execution.arrow.pyspark.enabled=true --database tpcds_parquet -f file:///home/hive/hive-testbench/spark-queries-tpcds/query42.sql

---

Performance Tuning – Core Gluten Parameters

1. Memory Configuration

• spark.memory.offHeap.enabled – enable off-heap memory
• spark.memory.offHeap.size – off-heap memory size (recommend 8–12g per executor)
• spark.gluten.memory.fraction – fraction of off-heap for Gluten (recommend ~0.7)

Rule of thumb

2. Shuffle Configuration

• spark.shuffle.manager = org.apache.spark.shuffle.sort.ColumnarShuffleManager
• spark.gluten.sql.columnar.shuffle.codec = lz4 (or zstd)

3. Velox Backend Settings

Key parameters to tune:

• spark.gluten.sql.columnar.backend.velox.maxBatchSize = 16384–32768
• spark.gluten.sql.columnar.backend.velox.memCacheSize = 2–4g
• spark.gluten.sql.columnar.backend.velox.aggregationPreferredSize = 1048576–4194304
• spark.gluten.sql.columnar.backend.velox.bloomFilterEnabled = true (join-heavy workloads)

4. Spilling Configuration

--conf spark.gluten.sql.columnar.backend.velox.spillEnabled=true --conf spark.gluten.sql.columnar.backend.velox.spillPath=/mnt/ssd/velox-spill

5. Adaptive Query Execution (AQE)

--conf spark.sql.adaptive.enabled=true
--conf spark.sql.adaptive.coalescePartitions.enabled=true
--conf spark.sql.adaptive.skewJoin.enabled=true
--conf spark.sql.adaptive.localShuffleReader.enabled=true
--conf spark.sql.adaptive.advisoryPartitionSizeInBytes=128MB

---

Monitoring and Validation

Verify Gluten is Active

Method 1 – Spark UI

• Open Spark UI: http://<driver-host>:4040
• Environment tab → check:
• spark.plugins = org.apache.gluten.GlutenPlugin
• spark.gluten.sql.enabled = true

Method 2 – Query Plan

val df = spark.sql("SELECT * FROM large_table WHERE id > 100")
df.explain(true)
// Look for: GlutenColumnarToRow, GlutenScan, GlutenFilter, GlutenProject

df = spark.sql("SELECT * FROM large_table WHERE id > 100")
df.explain(True)
# Look for Gluten operators in the physical plan

Method 3 – Logs

grep -i "gluten" /var/log/spark/spark-driver.log
# "GlutenPlugin is loaded"
# "Velox backend is initialized"

Performance Metrics

• Query execution time (Spark UI → SQL tab)
• Shuffle read/write size and time
• GC time
• Off-heap memory usage and spilling
• CPU utilization

Example comparison

# Without Gluten
spark-sql --database mydb -e "SELECT COUNT(*) FROM large_table"
# With Gluten
spark-sql --jars gluten-velox-bundle-*.jar --conf spark.plugins=org.apache.gluten.GlutenPlugin --conf spark.memory.offHeap.enabled=true --conf spark.memory.offHeap.size=8g --conf spark.gluten.sql.enabled=true --database mydb -e "SELECT COUNT(*) FROM large_table"

---

Benchmark Your Workload

#!/bin/bash
# Baseline (without Gluten)
echo "=== Running WITHOUT Gluten ==="
time spark-sql --database mydb -f my_query.sql > baseline.out

# With Gluten
echo "=== Running WITH Gluten ==="
time spark-sql --jars /usr/odp/current/spark3-client/jars/gluten-velox-bundle-*.jar --conf spark.plugins=org.apache.gluten.GlutenPlugin --conf spark.memory.offHeap.enabled=true --conf spark.memory.offHeap.size=8g --conf spark.gluten.sql.enabled=true --database mydb -f my_query.sql > gluten.out

# Compare results
diff baseline.out gluten.out # Should be identical

---

Best Practices

1. Data Format Recommendations

Best performance

• Parquet (columnar, compressed)
• ORC (columnar, compressed)

Good

• CSV (large files)
• JSON (structured)

Poor

• Small files (<128MB)
• Plain uncompressed text

-- Convert to Parquet for best results
CREATE TABLE optimized_table
USING PARQUET
AS SELECT * FROM original_table;

2. Partition Strategy

• Partition by date/time for time-series
• Aim for 128–256MB file/partition size
• Avoid thousands of tiny files

-- Good partitioning
CREATE TABLE events (
 id BIGINT,
 event_time TIMESTAMP,
 data STRING
)
USING PARQUET
PARTITIONED BY (date DATE)
LOCATION '/data/events';

3. Query Optimization

Do

• Use WHERE predicates for pushdown
• Use partition pruning
• SELECT only needed columns
• Enable bloom filters for joins

Avoid

• SELECT * on wide tables
• Complex nested UDFs
• Excessive small shuffles

4. Resource Allocation

# Small jobs (< 1TB)
--executor-memory 8g
--executor-cores 4
--conf spark.memory.offHeap.size=6g

# Medium jobs (1–10TB)
--executor-memory 16g
--executor-cores 4
--conf spark.memory.offHeap.size=12g

# Large jobs (> 10TB)
--executor-memory 32g
--executor-cores 8
--conf spark.memory.offHeap.size=24g

5. Fallback Strategy

# Disable Gluten if needed
--conf spark.gluten.sql.enabled=false
# Or omit Gluten JAR and plugin configs entirely

---

Troubleshooting

Issue 1: “Plugin GlutenPlugin could not be loaded”

Symptoms

ERROR PluginContainer: Error initializing plugin org.apache.gluten.GlutenPlugin

Fix

• Verify JAR path:

ls -lh /usr/odp/current/spark3-client/jars/gluten-velox-bundle-*.jar

• Ensure JAR is included in --jars
• Check Spark version compatibility (Gluten 1.4.0 ↔ Spark 3.5.x)

Issue 2: Native Library Load Failure

Symptoms

java.lang.UnsatisfiedLinkError: no velox in java.library.path

Fix

unzip -l gluten-velox-bundle-*.jar | grep "\.so$"
cat /proc/cpuinfo | grep avx2 # check AVX2 support

Issue 3: Out of Memory (OOM)

Fix

• Increase off-heap:

--conf spark.memory.offHeap.size=16g

• Enable spilling:

--conf spark.gluten.sql.columnar.backend.velox.spillEnabled=true --conf spark.gluten.sql.columnar.backend.velox.spillPath=/tmp/velox-spill

• Reduce batch size:

--conf spark.gluten.sql.columnar.backend.velox.maxBatchSize=16384

Issue 4: Result Mismatch vs Vanilla Spark

Fix

• Check data types (decimals, timestamps)
• Ensure query is supported by Gluten
• Temporarily disable Gluten:

spark.conf.set("spark.gluten.sql.enabled", "false")
// run query
spark.conf.set("spark.gluten.sql.enabled", "true")

Issue 5: Slower with Gluten

• Very small data (<100MB) → use vanilla Spark
• Excessive ColumnarToRow conversions → inspect plan and simplify query
• Increase spark.memory.offHeap.size
• Tune spark.gluten.sql.columnar.shuffle.codec (try lz4 or zstd)

---

Debugging Tips

Enable more logging:

--conf spark.gluten.sql.columnar.backend.velox.logLevel=INFO --conf spark.sql.adaptive.logLevel=DEBUG

Check plans:

val df = spark.sql("YOUR_QUERY")
df.explain(mode = "extended")
df.queryExecution.debug.codegen()

Inspect Gluten-related configs:

spark.sparkContext.getConf.getAll
 .filter(_._1.contains("gluten"))
 .foreach(println)

---

FAQ

General

Q: Do I need to rewrite my Spark applications? A: No. Gluten works transparently with existing Spark SQL, DataFrame, and Dataset code.

Q: Will query results be the same? A: Yes. If you see differences, treat them as bugs and escalate.

Q: Does Gluten work with PySpark / SparkR / Scala / Java? A: Yes.

Q: Does Gluten support UDFs? A: Built-in functions are supported; custom UDFs usually fall back to Spark.

Q: What about Structured Streaming? A: Works for SQL operations in micro-batch mode.

Performance

• Typical speedup: 2–5× on analytical workloads
• Limited gains on very small queries (<100MB)
• Fully compatible with dynamic allocation

Technical

• File formats: Parquet, ORC, CSV, JSON (best with Parquet / ORC)
• Works with Hive metastore and Hive tables
• Compatible with Delta Lake / Iceberg / Hudi through Spark
• Storage: S3, HDFS, ADLS, GCS, etc.

Disable for a specific query:

spark.conf.set("spark.gluten.sql.enabled", "false")
// run query
spark.conf.set("spark.gluten.sql.enabled", "true")

Compatibility & Deployment

• Spark versions: 3.3.x, 3.4.x, 3.5.x (Gluten 1.4.0)
• Java: 8 and 11
• Cluster managers: YARN, Kubernetes, Standalone
• Arch: x86_64 (ARM64 planned)

Rollback: remove Gluten JAR + plugin configs. Works in notebooks (Jupyter, Zeppelin) via Spark conf.

---

Quick Reference

Essential Config

# Minimum
--jars gluten-velox-bundle-*.jar
--conf spark.plugins=org.apache.gluten.GlutenPlugin
--conf spark.memory.offHeap.enabled=true
--conf spark.memory.offHeap.size=8g
--conf spark.gluten.sql.enabled=true
--conf spark.gluten.sql.columnar.backend.lib=velox

# Recommended
--conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager
--conf spark.sql.adaptive.enabled=true
--conf spark.gluten.sql.columnar.backend.velox.spillEnabled=true
# JDK 11 only
--conf spark.driver.extraJavaOptions="--add-opens java.base/java.lang=ALL-UNNAMED"
--conf spark.executor.extraJavaOptions="--add-opens java.base/java.lang=ALL-UNNAMED"

Quick Checks

# Plugin loaded?
grep "GlutenPlugin" spark-driver.log

# Plan shows Gluten operators?
spark.sql("YOUR_QUERY").explain()
# Look for GlutenScan / GlutenFilter / GlutenProject

---

Appendix – Key Gluten Parameters

• spark.gluten.sql.enabled – master enable/disable switch
• spark.gluten.sql.columnar.backend.lib – backend engine (velox)
• spark.gluten.memory.fraction – fraction of off-heap for Gluten
• spark.gluten.sql.columnar.preferColumnar – prefer columnar exec
• spark.gluten.sql.columnar.forceShuffledHashJoin – force hash join
• spark.gluten.sql.columnar.shuffle.codec – shuffle codec (lz4)
• spark.gluten.sql.columnar.backend.velox.maxBatchSize – rows per batch
• spark.gluten.sql.columnar.backend.velox.memCacheSize – cache size
• spark.gluten.sql.columnar.backend.velox.spillEnabled – enable spilling
• spark.gluten.sql.columnar.backend.velox.spillPath – spill directory
• spark.gluten.sql.columnar.backend.velox.bloomFilterEnabled – bloom filters
• spark.gluten.sql.columnar.backend.velox.aggregationPreferredSize – aggregation table size

---

Support and Resources

• Apache Gluten GitHub – project source and issues
• Velox GitHub – execution engine documentation
• Apache Spark Tuning Guide – general Spark performance best practices