Troubleshoot Spark 3 Dynamic Allocation and Shuffle Service Issues After ODP Upgrade

After upgrading from Spark2 to Spark3 “on ODP”, Spark applications may fail when dynamic allocation and external shuffle service are enabled.

Symptoms can include:

• Spark jobs hanging
• Executor allocation failures
• Shuffle-related exceptions
• PySpark startup failures
• Spark service check failures

Symptoms

Applications run successfully without dynamic allocation:

xxxxxxxxxx

spark.dynamicAllocation.enabled=false

but fail when the following settings are enabled:

xxxxxxxxxx

spark.dynamicAllocation.enabled=true

spark.dynamicAllocation.minExecutors

spark.dynamicAllocation.maxExecutors

spark.shuffle.service.enabled=true

Cause

During Spark2 → Spark3 migration, shuffle service configuration may not fully align with the Spark3 deployment.

Common causes include:

Incorrect Shuffle Classpath

xxxxxxxxxx

yarn.nodemanager.aux-services.spark3_shuffle.classpath

does not point to the Spark3 shuffle libraries.

Incorrect Shuffle Service Port

xxxxxxxxxx

spark.shuffle.service.port

is configured with a non-functional port.

Legacy Spark2 Components

Spark2 symlinks remain present and interfere with Spark3 client execution.

Resolution

Update Shuffle Port

Configure:

xxxxxxxxxx

spark.shuffle.service.port=7337

Configure the location for jar files for the external shuffle service yarn.nodemanager.aux-services.spar.

xxxxxxxxxx

yarn.nodemanager.aux-services.spark3_shuffle.classpath=/usr/odp/current/spark3-client/aux/*

Restart:

xxxxxxxxxx

Spark Services

YARN Services

Validate Dynamic Allocation

Re-enable:

xxxxxxxxxx

spark.dynamicAllocation.enabled=true

spark.shuffle.service.enabled=true

Submit a Spark application and verify that executor allocation functions normally.

PySpark Failure After Spark3 Migration

Symptoms

xxxxxxxxxx

Multiple versions of Spark are installed

but SPARK_MAJOR_VERSION is not set

followed by:

xxxxxxxxxx

TypeError: code() argument 13 must be str, not int

Cause

PySpark is launching the Spark2 runtime instead of Spark3.

Resolution

• Verify Spark2 is no longer required.
• Remove obsolete Spark2 references:

xxxxxxxxxx

spark2-client

spark2-historyserver

spark2-thriftserver

Restart Spark services.

Validate:

xxxxxxxxxx

spark-shell

and

xxxxxxxxxx

pyspark

start successfully.

Validation

Run:

xxxxxxxxxx

spark-shell

Run:

xxxxxxxxxx

pyspark

Submit:

xxxxxxxxxx

spark-submit

with dynamic allocation enabled.

Confirm:

• Executors are allocated successfully.
• Shuffle operations complete successfully.
• No Livy or YARN errors are reported.

Best Practices

• Remove obsolete Spark2 components after migration.
• Verify shuffle service configuration before enabling dynamic allocation.
• Validate Spark shell, PySpark, and Spark submit workflows after upgrade.
• Test production jobs before enabling dynamic allocation in production.

Summary

Spark3 upgrades may expose issues related to external shuffle services, dynamic allocation, and legacy Spark2 references.

Correcting the Spark3 shuffle configuration, validating the shuffle service port, and removing obsolete Spark2 components typically resolve these issues and restore normal Spark operation.