Gluten with Velox

Apache Gluten with Velox backend is a native execution plugin for Apache Spark that accelerates SQL query performance without requiring any code changes to your existing Spark applications.

Key Benefits

Zero Code Changes – Works with existing Spark SQL, DataFrame, and Dataset APIs
Significant Performance Gains – 2–5× faster on analytical workloads
Drop-in Replacement – Simply add JAR and configuration
Transparent Acceleration – Automatically offloads supported operations to native Velox engine
Production Ready – Battle-tested on TPC-DS, TPC-H benchmarks

What is Gluten?

Apache Gluten is a Spark plugin that offloads SQL query execution from JVM to native C++ execution engines. It acts as a “glue” layer between Apache Spark and vectorized execution engines like Velox.

Architecture Overview

┌─────────────────────────────────────────────────┐ │ Your Spark Application │ │ (SQL, DataFrame, Dataset APIs) │ │ NO CHANGES NEEDED │ └─────────────────────────────────────────────────┘ ↓ ┌─────────────────────────────────────────────────┐ │ Apache Spark SQL Engine │ │ (Query Planning & Optimization) │ └─────────────────────────────────────────────────┘ ↓ ┌─────────────────────────────────────────────────┐ │ Gluten Plugin Layer │ │ (Translates Spark Plans to Native Plans) │ └─────────────────────────────────────────────────┘ ↓ ┌─────────────────────────────────────────────────┐ │ Velox Execution Engine (C++) │ │ (Vectorized Processing, SIMD, Zero-Copy) │ └─────────────────────────────────────────────────┘

Why Velox?

Velox is Meta's unified execution engine that powers:

Meta's data warehouse (Presto)
Stream processing systems
Machine learning infrastructure

Key Features

Vectorized execution – processes data in batches
SIMD instructions – hardware-accelerated operations
Advanced codegen – runtime code generation
Memory efficiency – Arrow-based zero-copy design

How It Works

Transparent Query Acceleration

When you enable Gluten, Spark automatically:

Analyzes your SQL query plan
Identifies operations that can be accelerated (scans, filters, joins, aggregations)
Offloads those operations to the Velox native engine
Falls back to Spark for unsupported operations
Returns results – your application receives the same output, just faster!

Supported Operations

✅ Fully Supported

Table scans (Parquet, ORC, CSV)
Filters and predicates
Projections
Hash joins (inner, left, right, full outer, semi, anti)
Hash aggregations (sum, avg, min, max, count)
Sort operations
Window functions
String operations
Date/time operations
Math functions

⚠️ Partial Support

Some complex UDFs
Certain window function combinations
Specialized data types

❌ Not Supported

Custom data sources
Complex nested UDFs
Some exotic data types

Performance Expectations

Based on TPC-DS and production workloads:

Workload Type	Expected Speedup
Scan-heavy queries	2–3×
Join-heavy queries	3–5×
Aggregation-heavy	2–4×
Complex analytics	2–3×
Simple queries	1.5–2×

Real-world Examples

TPC-DS Query 72: 45s → 12s (3.75× speedup)
TPC-DS Query 95: 120s → 35s (3.4× speedup)

Getting Started

Prerequisites

Component	Version
Apache Spark	3.5.x
Java	8 or 11 (upcoming release)
Operating System	Linux x86_64
CPU	AVX2 support recommended

Step 1: Obtain Gluten JAR

Contact your Acceldata account manager or download from:

gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.4.0.jar

Place JAR in:

/usr/odp/current/spark3-client/jars/

Step 2: Enable Gluten (No Code Changes!)

Option A: Via `spark-submit`

Option B: Via `spark-defaults.conf`

Add to $SPARK_HOME/conf/spark-defaults.conf:

spark.jars /usr/odp/current/spark3-client/jars/gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.4.0.jar spark.plugins org.apache.gluten.GlutenPlugin spark.memory.offHeap.enabled true spark.memory.offHeap.size 8g spark.gluten.sql.enabled true spark.gluten.sql.columnar.backend.lib velox

Option C: Programmatic Configuration (Scala/Java)

val spark = SparkSession.builder() .appName("MyApp") .config("spark.plugins", "org.apache.gluten.GlutenPlugin") .config("spark.memory.offHeap.enabled", "true") .config("spark.memory.offHeap.size", "8g") .config("spark.gluten.sql.enabled", "true") .config("spark.gluten.sql.columnar.backend.lib", "velox") .getOrCreate() // Your existing code runs unchanged! spark.sql("SELECT * FROM large_table WHERE value > 100").show()

Option D: PySpark

from pyspark.sql import SparkSession spark = SparkSession.builder .appName("MyApp") .config("spark.plugins", "org.apache.gluten.GlutenPlugin") .config("spark.memory.offHeap.enabled", "true") .config("spark.memory.offHeap.size", "8g") .config("spark.gluten.sql.enabled", "true") .config("spark.gluten.sql.columnar.backend.lib", "velox") .getOrCreate() # Your existing code runs unchanged! df = spark.sql("SELECT * FROM large_table WHERE value > 100") df.show()

Step 3: Run Your Application

That's it! Your application runs exactly as before, but faster.

Configuration Guide

Minimal Configuration (Quick Start)

For JDK 11

spark-submit --jars /usr/odp/current/spark3-client/jars/gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.4.0.jar --conf spark.plugins=org.apache.gluten.GlutenPlugin --conf spark.memory.offHeap.enabled=true --conf spark.memory.offHeap.size=8g --conf spark.gluten.sql.enabled=true --conf spark.gluten.sql.columnar.backend.lib=velox --conf spark.driver.extraJavaOptions="--add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED" --conf spark.executor.extraJavaOptions="--add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED" your-application.jar

For JDK 8

Info

Note: JDK 8 does not require the --add-opens parameters.

Recommended Production Configuration

spark-submit --master yarn --deploy-mode cluster --driver-memory 4g --executor-memory 16g --executor-cores 4 --num-executors 10 --jars /usr/odp/current/spark3-client/jars/gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.4.0.jar # Core Gluten Settings --conf spark.plugins=org.apache.gluten.GlutenPlugin --conf spark.gluten.sql.enabled=true --conf spark.gluten.sql.columnar.backend.lib=velox # Memory Configuration --conf spark.memory.offHeap.enabled=true --conf spark.memory.offHeap.size=12g --conf spark.gluten.memory.fraction=0.7 # Shuffle Configuration --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager --conf spark.gluten.sql.columnar.shuffle.codec=lz4 # Adaptive Query Execution --conf spark.sql.adaptive.enabled=true --conf spark.sql.adaptive.coalescePartitions.enabled=true --conf spark.sql.adaptive.skewJoin.enabled=true # Velox-Specific Settings --conf spark.gluten.sql.columnar.backend.velox.spillEnabled=true --conf spark.gluten.sql.columnar.backend.velox.spillPath=/tmp/velox-spill # Java Options (JDK 11 only) --conf spark.driver.extraJavaOptions="--add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED" --conf spark.executor.extraJavaOptions="--add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED" your-application.jar

TPC-DS Benchmark Configuration (Tested)

This is the exact configuration used by Acceldata for TPC-DS benchmarking.

For JDK 11

spark-sql --driver-memory 3g --executor-memory 12g --executor-cores 3 --num-executors 9 --jars /usr/odp/current/spark3-client/jars/gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.4.0.jar --conf spark.plugins=org.apache.gluten.GlutenPlugin --conf spark.sql.optimizer.runtime.bloomFilter.applicationSideScanSizeThreshold=128MB --conf spark.memory.offHeap.enabled=true --conf spark.memory.offHeap.size=8g --conf spark.gluten.sql.enabled=true --conf spark.gluten.sql.columnar.backend.lib=velox --conf spark.gluten.sql.columnar.backend.velox=true --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager --conf spark.gluten.sql.columnar.shuffle.codec=lz4 --conf spark.driver.extraJavaOptions="--illegal-access=permit -Dio.netty.tryReflectionSetAccessible=true --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED" --conf spark.executor.extraJavaOptions="--illegal-access=permit -Dio.netty.tryReflectionSetAccessible=true --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED" --conf spark.sql.adaptive.enabled=true --conf spark.sql.adaptive.coalescePartitions.enabled=true --conf spark.sql.adaptive.skewJoin.enabled=true --conf spark.sql.adaptive.localShuffleReader.enabled=true --conf spark.gluten.sql.columnar.backend.velox.memCacheSize=4g --conf spark.sql.inMemoryColumnarStorage.compressed=true --conf spark.sql.inMemoryColumnarStorage.batchSize=50000 --conf spark.gluten.sql.columnar.forceShuffledHashJoin=false --conf spark.gluten.memory.fraction=0.7 --conf spark.gluten.sql.columnar.backend.velox.maxBatchSize=32768 --conf spark.gluten.sql.columnar.backend.velox.bloomFilterEnabled=true --conf spark.gluten.sql.columnar.preferColumnar=true --conf spark.gluten.sql.columnar.backend.velox.aggregationPreferredSize=1048576 --conf spark.sql.adaptive.advisoryPartitionSizeInBytes=128MB --conf spark.sql.adaptive.maxShuffledHashJoinLocalMapThreshold=1GB --conf spark.gluten.sql.columnar.backend.velox.spillEnabled=true --conf spark.gluten.sql.columnar.backend.velox.spillPath=/tmp/velox-spill --conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.sql.execution.arrow.pyspark.enabled=true --database tpcds_parquet -f file:///home/hive/hive-testbench/spark-queries-tpcds/query42.sql

For JDK 8

spark-sql --driver-memory 3g --executor-memory 12g --executor-cores 3 --num-executors 9 --jars /usr/odp/current/spark3-client/jars/gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.4.0.jar --conf spark.plugins=org.apache.gluten.GlutenPlugin --conf spark.sql.optimizer.runtime.bloomFilter.applicationSideScanSizeThreshold=128MB --conf spark.memory.offHeap.enabled=true --conf spark.memory.offHeap.size=8g --conf spark.gluten.sql.enabled=true --conf spark.gluten.sql.columnar.backend.lib=velox --conf spark.gluten.sql.columnar.backend.velox=true --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager --conf spark.gluten.sql.columnar.shuffle.codec=lz4 --conf spark.sql.adaptive.enabled=true --conf spark.sql.adaptive.coalescePartitions.enabled=true --conf spark.sql.adaptive.skewJoin.enabled=true --conf spark.sql.adaptive.localShuffleReader.enabled=true --conf spark.gluten.sql.columnar.backend.velox.memCacheSize=4g --conf spark.sql.inMemoryColumnarStorage.compressed=true --conf spark.sql.inMemoryColumnarStorage.batchSize=50000 --conf spark.gluten.sql.columnar.forceShuffledHashJoin=false --conf spark.gluten.memory.fraction=0.7 --conf spark.gluten.sql.columnar.backend.velox.maxBatchSize=32768 --conf spark.gluten.sql.columnar.backend.velox.bloomFilterEnabled=true --conf spark.gluten.sql.columnar.preferColumnar=true --conf spark.gluten.sql.columnar.backend.velox.aggregationPreferredSize=1048576 --conf spark.sql.adaptive.advisoryPartitionSizeInBytes=128MB --conf spark.sql.adaptive.maxShuffledHashJoinLocalMapThreshold=1GB --conf spark.gluten.sql.columnar.backend.velox.spillEnabled=true --conf spark.gluten.sql.columnar.backend.velox.spillPath=/tmp/velox-spill --conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.sql.execution.arrow.pyspark.enabled=true --database tpcds_parquet -f file:///home/hive/hive-testbench/spark-queries-tpcds/query42.sql

Performance Tuning – Core Gluten Parameters

1. Memory Configuration

spark.memory.offHeap.enabled – enable off-heap memory
spark.memory.offHeap.size – off-heap memory size (recommend 8–12g per executor)
spark.gluten.memory.fraction – fraction of off-heap for Gluten (recommend ~0.7)

Rule of thumb

Info

Off-heap memory ≈ 60–75% of executor memory Example: executor-memory=16g → offHeap.size=10–12g

2. Shuffle Configuration

spark.shuffle.manager = org.apache.spark.shuffle.sort.ColumnarShuffleManager
spark.gluten.sql.columnar.shuffle.codec = lz4 (or zstd)

3. Velox Backend Settings

Key parameters to tune:

spark.gluten.sql.columnar.backend.velox.maxBatchSize = 16384–32768
spark.gluten.sql.columnar.backend.velox.memCacheSize = 2–4g
spark.gluten.sql.columnar.backend.velox.aggregationPreferredSize = 1048576–4194304
spark.gluten.sql.columnar.backend.velox.bloomFilterEnabled = true (join-heavy workloads)

4. Spilling Configuration

--conf spark.gluten.sql.columnar.backend.velox.spillEnabled=true --conf spark.gluten.sql.columnar.backend.velox.spillPath=/mnt/ssd/velox-spill

5. Adaptive Query Execution (AQE)

--conf spark.sql.adaptive.enabled=true --conf spark.sql.adaptive.coalescePartitions.enabled=true --conf spark.sql.adaptive.skewJoin.enabled=true --conf spark.sql.adaptive.localShuffleReader.enabled=true --conf spark.sql.adaptive.advisoryPartitionSizeInBytes=128MB

Monitoring and Validation

Verify Gluten is Active

Method 1 – Spark UI

Open Spark UI: http://<driver-host>:4040
Environment tab → check:
spark.plugins = org.apache.gluten.GlutenPlugin
spark.gluten.sql.enabled = true

Method 2 – Query Plan

val df = spark.sql("SELECT * FROM large_table WHERE id > 100") df.explain(true) // Look for: GlutenColumnarToRow, GlutenScan, GlutenFilter, GlutenProject

df = spark.sql("SELECT * FROM large_table WHERE id > 100") df.explain(True) # Look for Gluten operators in the physical plan

Method 3 – Logs

grep -i "gluten" /var/log/spark/spark-driver.log # "GlutenPlugin is loaded" # "Velox backend is initialized"

Performance Metrics

Query execution time (Spark UI → SQL tab)
Shuffle read/write size and time
GC time
Off-heap memory usage and spilling
CPU utilization

Example comparison

# Without Gluten spark-sql --database mydb -e "SELECT COUNT(*) FROM large_table" # With Gluten spark-sql --jars gluten-velox-bundle-*.jar --conf spark.plugins=org.apache.gluten.GlutenPlugin --conf spark.memory.offHeap.enabled=true --conf spark.memory.offHeap.size=8g --conf spark.gluten.sql.enabled=true --database mydb -e "SELECT COUNT(*) FROM large_table"

Benchmark Your Workload

#!/bin/bash # Baseline (without Gluten) echo "=== Running WITHOUT Gluten ===" time spark-sql --database mydb -f my_query.sql > baseline.out # With Gluten echo "=== Running WITH Gluten ===" time spark-sql --jars /usr/odp/current/spark3-client/jars/gluten-velox-bundle-*.jar --conf spark.plugins=org.apache.gluten.GlutenPlugin --conf spark.memory.offHeap.enabled=true --conf spark.memory.offHeap.size=8g --conf spark.gluten.sql.enabled=true --database mydb -f my_query.sql > gluten.out # Compare results diff baseline.out gluten.out # Should be identical

Best Practices

1. Data Format Recommendations

Best performance

Parquet (columnar, compressed)
ORC (columnar, compressed)

Good

CSV (large files)
JSON (structured)

Poor

Small files (<128MB)
Plain uncompressed text

-- Convert to Parquet for best results CREATE TABLE optimized_table USING PARQUET AS SELECT * FROM original_table;

2. Partition Strategy

Partition by date/time for time-series
Aim for 128–256MB file/partition size
Avoid thousands of tiny files

-- Good partitioning CREATE TABLE events ( id BIGINT, event_time TIMESTAMP, data STRING ) USING PARQUET PARTITIONED BY (date DATE) LOCATION '/data/events';

3. Query Optimization

Use WHERE predicates for pushdown
Use partition pruning
SELECT only needed columns
Enable bloom filters for joins

Avoid

SELECT * on wide tables
Complex nested UDFs
Excessive small shuffles

4. Resource Allocation

# Small jobs (< 1TB) --executor-memory 8g --executor-cores 4 --conf spark.memory.offHeap.size=6g # Medium jobs (1–10TB) --executor-memory 16g --executor-cores 4 --conf spark.memory.offHeap.size=12g # Large jobs (> 10TB) --executor-memory 32g --executor-cores 8 --conf spark.memory.offHeap.size=24g

5. Fallback Strategy

# Disable Gluten if needed --conf spark.gluten.sql.enabled=false # Or omit Gluten JAR and plugin configs entirely

Troubleshooting

Issue 1: “Plugin GlutenPlugin could not be loaded”

Symptoms

ERROR PluginContainer: Error initializing plugin org.apache.gluten.GlutenPlugin

Fix

Verify JAR path:

ls -lh /usr/odp/current/spark3-client/jars/gluten-velox-bundle-*.jar

Ensure JAR is included in --jars
Check Spark version compatibility (Gluten 1.4.0 ↔ Spark 3.5.x)

Issue 2: Native Library Load Failure

Symptoms

java.lang.UnsatisfiedLinkError: no velox in java.library.path

Fix

unzip -l gluten-velox-bundle-*.jar | grep "\.so$" cat /proc/cpuinfo | grep avx2 # check AVX2 support

Issue 3: Out of Memory (OOM)

Fix

Increase off-heap:

--conf spark.memory.offHeap.size=16g

Enable spilling:

--conf spark.gluten.sql.columnar.backend.velox.spillEnabled=true --conf spark.gluten.sql.columnar.backend.velox.spillPath=/tmp/velox-spill

Reduce batch size:

--conf spark.gluten.sql.columnar.backend.velox.maxBatchSize=16384

Issue 4: Result Mismatch vs Vanilla Spark

Fix

Check data types (decimals, timestamps)
Ensure query is supported by Gluten
Temporarily disable Gluten:

spark.conf.set("spark.gluten.sql.enabled", "false") // run query spark.conf.set("spark.gluten.sql.enabled", "true")

Issue 5: Slower with Gluten

Very small data (<100MB) → use vanilla Spark
Excessive ColumnarToRow conversions → inspect plan and simplify query
Increase spark.memory.offHeap.size
Tune spark.gluten.sql.columnar.shuffle.codec (try lz4 or zstd)

Debugging Tips

Enable more logging:

--conf spark.gluten.sql.columnar.backend.velox.logLevel=INFO --conf spark.sql.adaptive.logLevel=DEBUG

Check plans:

val df = spark.sql("YOUR_QUERY") df.explain(mode = "extended") df.queryExecution.debug.codegen()

Inspect Gluten-related configs:

spark.sparkContext.getConf.getAll .filter(_._1.contains("gluten")) .foreach(println)

FAQ

General

Q: Do I need to rewrite my Spark applications? A: No. Gluten works transparently with existing Spark SQL, DataFrame, and Dataset code.

Q: Will query results be the same? A: Yes. If you see differences, treat them as bugs and escalate.

Q: Does Gluten work with PySpark / SparkR / Scala / Java? A: Yes.

Q: Does Gluten support UDFs? A: Built-in functions are supported; custom UDFs usually fall back to Spark.

Q: What about Structured Streaming? A: Works for SQL operations in micro-batch mode.

Performance

Typical speedup: 2–5× on analytical workloads
Limited gains on very small queries (<100MB)
Fully compatible with dynamic allocation

Technical

File formats: Parquet, ORC, CSV, JSON (best with Parquet / ORC)
Works with Hive metastore and Hive tables
Compatible with Delta Lake / Iceberg / Hudi through Spark
Storage: S3, HDFS, ADLS, GCS, etc.

Disable for a specific query:

spark.conf.set("spark.gluten.sql.enabled", "false") // run query spark.conf.set("spark.gluten.sql.enabled", "true")

Compatibility & Deployment

Spark versions: 3.3.x, 3.4.x, 3.5.x (Gluten 1.4.0)
Java: 8 and 11
Cluster managers: YARN, Kubernetes, Standalone
Arch: x86_64 (ARM64 planned)

Rollback: remove Gluten JAR + plugin configs. Works in notebooks (Jupyter, Zeppelin) via Spark conf.

Quick Reference

Essential Config

# Minimum --jars gluten-velox-bundle-*.jar --conf spark.plugins=org.apache.gluten.GlutenPlugin --conf spark.memory.offHeap.enabled=true --conf spark.memory.offHeap.size=8g --conf spark.gluten.sql.enabled=true --conf spark.gluten.sql.columnar.backend.lib=velox # Recommended --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager --conf spark.sql.adaptive.enabled=true --conf spark.gluten.sql.columnar.backend.velox.spillEnabled=true # JDK 11 only --conf spark.driver.extraJavaOptions="--add-opens java.base/java.lang=ALL-UNNAMED" --conf spark.executor.extraJavaOptions="--add-opens java.base/java.lang=ALL-UNNAMED"

Quick Checks

# Plugin loaded? grep "GlutenPlugin" spark-driver.log # Plan shows Gluten operators? spark.sql("YOUR_QUERY").explain() # Look for GlutenScan / GlutenFilter / GlutenProject

Appendix – Key Gluten Parameters

spark.gluten.sql.enabled – master enable/disable switch
spark.gluten.sql.columnar.backend.lib – backend engine (velox)
spark.gluten.memory.fraction – fraction of off-heap for Gluten
spark.gluten.sql.columnar.preferColumnar – prefer columnar exec
spark.gluten.sql.columnar.forceShuffledHashJoin – force hash join
spark.gluten.sql.columnar.shuffle.codec – shuffle codec (lz4)
spark.gluten.sql.columnar.backend.velox.maxBatchSize – rows per batch
spark.gluten.sql.columnar.backend.velox.memCacheSize – cache size
spark.gluten.sql.columnar.backend.velox.spillEnabled – enable spilling
spark.gluten.sql.columnar.backend.velox.spillPath – spill directory
spark.gluten.sql.columnar.backend.velox.bloomFilterEnabled – bloom filters
spark.gluten.sql.columnar.backend.velox.aggregationPreferredSize – aggregation table size

Support and Resources

Apache Gluten GitHub – project source and issues
Velox GitHub – execution engine documentation
Apache Spark Tuning Guide – general Spark performance best practices

Last updated on Nov 20, 2025

Was this page helpful?

Gluten with Velox

Key Benefits

What is Gluten?

Architecture Overview

Why Velox?

Key Features

How It Works

Transparent Query Acceleration

Supported Operations

✅ Fully Supported

⚠️ Partial Support

❌ Not Supported

Performance Expectations

Getting Started

Prerequisites

Step 1: Obtain Gluten JAR

Step 2: Enable Gluten (No Code Changes!)

Option A: Via spark-submit

Option B: Via spark-defaults.conf

Option C: Programmatic Configuration (Scala/Java)

Option D: PySpark

Step 3: Run Your Application

Configuration Guide

Minimal Configuration (Quick Start)

For JDK 11

For JDK 8

Recommended Production Configuration

TPC-DS Benchmark Configuration (Tested)

For JDK 11

For JDK 8

Performance Tuning – Core Gluten Parameters

1. Memory Configuration

2. Shuffle Configuration

3. Velox Backend Settings

4. Spilling Configuration

5. Adaptive Query Execution (AQE)

Monitoring and Validation

Verify Gluten is Active

Performance Metrics

Benchmark Your Workload

Best Practices

1. Data Format Recommendations

2. Partition Strategy

3. Query Optimization

4. Resource Allocation

5. Fallback Strategy

Troubleshooting

Issue 1: “Plugin GlutenPlugin could not be loaded”

Issue 2: Native Library Load Failure

Issue 3: Out of Memory (OOM)

Issue 4: Result Mismatch vs Vanilla Spark

Issue 5: Slower with Gluten

Debugging Tips

FAQ

General

Performance

Technical

Compatibility & Deployment

Quick Reference

Essential Config

Quick Checks

Appendix – Key Gluten Parameters

Support and Resources

Option A: Via `spark-submit`

Option B: Via `spark-defaults.conf`