Troubleshoot Spark 3 Dynamic Allocation and Shuffle Service Issues After ODP Upgrade

After upgrading from Spark2 to Spark3 “on ODP”, Spark applications may fail when dynamic allocation and external shuffle service are enabled.

Symptoms can include:

  • Spark jobs hanging

  • Executor allocation failures

  • Shuffle-related exceptions

  • PySpark startup failures

  • Spark service check failures

Symptoms

Applications run successfully without dynamic allocation:

spark.dynamicAllocation.enabled=false

but fail when the following settings are enabled:

spark.dynamicAllocation.enabled=true spark.dynamicAllocation.minExecutors spark.dynamicAllocation.maxExecutors spark.shuffle.service.enabled=true

Cause

During Spark2 → Spark3 migration, shuffle service configuration may not fully align with the Spark3 deployment.

Common causes include:

Incorrect Shuffle Classpath

yarn.nodemanager.aux-services.spark3_shuffle.classpath

does not point to the Spark3 shuffle libraries.

Incorrect Shuffle Service Port

spark.shuffle.service.port

is configured with a non-functional port.

Legacy Spark2 Components

Spark2 symlinks remain present and interfere with Spark3 client execution.

Resolution

Update Shuffle Port

Configure:

spark.shuffle.service.port=7337

Configure the location for jar files for the external shuffle service yarn.nodemanager.aux-services.spar.

yarn.nodemanager.aux-services.spark3_shuffle.classpath=/usr/odp/current/spark3-client/aux/*

Restart:

Spark Services YARN Services

Validate Dynamic Allocation

Re-enable:

spark.dynamicAllocation.enabled=true spark.shuffle.service.enabled=true

Submit a Spark application and verify that executor allocation functions normally.

PySpark Failure After Spark3 Migration

Symptoms

Multiple versions of Spark are installed but SPARK_MAJOR_VERSION is not set

followed by:

TypeError: code() argument 13 must be str, not int

Cause

PySpark is launching the Spark2 runtime instead of Spark3.

Resolution

  • Verify Spark2 is no longer required.

  • Remove obsolete Spark2 references:

spark2-client spark2-historyserver spark2-thriftserver

Restart Spark services.

Validate:

spark-shell

and

pyspark

start successfully.

Validation

Run:

spark-shell

Run:

pyspark

Submit:

spark-submit

with dynamic allocation enabled.

Confirm:

  • Executors are allocated successfully.

  • Shuffle operations complete successfully.

  • No Livy or YARN errors are reported.

Best Practices

  • Remove obsolete Spark2 components after migration.

  • Verify shuffle service configuration before enabling dynamic allocation.

  • Validate Spark shell, PySpark, and Spark submit workflows after upgrade.

  • Test production jobs before enabling dynamic allocation in production.

Summary

Spark3 upgrades may expose issues related to external shuffle services, dynamic allocation, and legacy Spark2 references.

Correcting the Spark3 shuffle configuration, validating the shuffle service port, and removing obsolete Spark2 components typically resolve these issues and restore normal Spark operation.


null
On This Page
Troubleshoot Spark 3 Dynamic Allocation and Shuffle Service Issues After ODP UpgradeSymptomsCauseResolutionValidate Dynamic AllocationPySpark Failure After Spark3 MigrationValidationBest PracticesSummaryIncorrect Shuffle ClasspathIncorrect Shuffle Service PortLegacy Spark2 ComponentsUpdate Shuffle Port