Spark 4.1.1 Shuffle Service Limitation on JDK 11

Spark 4.1.1 includes an external YARN shuffle service named spark_shuffle_411.

The spark_shuffle_411 service requires Java 17. If YARN NodeManager runs on JDK 11 and attempts to load this service, NodeManager fails to start.

Affected Environments

The issue affects ODP clusters that meet the following conditions:

You can install and run Spark 4.1.1 applications on a cluster where NodeManager runs on JDK 11, provided that spark_shuffle_411 isn't enabled.

Spark 4.1.1 application components, such as the application master and executors, can use JDK 17 independently of the NodeManager Java version.

When spark_shuffle_411 is disabled, Spark 4.1.1 applications use the standard Spark shuffle mechanism instead of the Spark 4 external NodeManager shuffle service.

Cause

When SPARK4 is added through Ambari, spark_shuffle_411 might be added to:

NodeManager attempts to load every configured auxiliary service during startup.

Because spark_shuffle_411 is compiled for Java 17, NodeManager running on JDK 11 can't load it and fails to start.

NodeManager Error

The NodeManager log contains an error similar to the following:

The class file versions correspond to the following Java versions:

• Class file version 61 — Java 17

• Class file version 55 — Java 11

Shuffle Service compatibility

Example Unsupported Configuration

The following configuration isn't supported when NodeManager runs on JDK 11:

In this configuration, spark_shuffle_411 prevents NodeManager from starting.

A configuration without the Spark 4.1.1 shuffle service can resemble:

Impact

If spark_shuffle_411 is enabled while NodeManager runs on JDK 11:

• NodeManager fails to start on affected hosts.

• YARN can have fewer or no healthy NodeManagers.

• YARN applications might not be able to run.

• Adding SPARK4 can affect the availability of the YARN layer if the unsupported shuffle service is enabled.

Workaround for JDK 11 NodeManagers

If NodeManager runs on JDK 11, remove spark_shuffle_411 from the YARN auxiliary services configuration.

1. In the Ambari UI, go to YARN > Configs.

2. Locate the yarn.nodemanager.aux-services property in yarn-site.

3. Remove spark_shuffle_411.

4. Keep the required Spark 3 shuffle services.

   For example:

   mapreduce_shuffle,spark_shuffle_355,{{timeline_collector}},spark_shuffle_333,spark_shuffle_351

   Or use the minimal required configuration:

   mapreduce_shuffle,spark_shuffle_355,{{timeline_collector}}

5. Save the configuration.

6. Restart the NodeManagers. Restart the ResourceManager if Ambari requires it.

7. Verify that the NodeManagers start successfully.

8. Check the NodeManager startup logs and confirm that the spark_shuffle_411 class loading error no longer appears.

You can keep Spark 4.1.1 installed and continue to submit Spark 4.1.1 applications. This workaround disables only the Spark 4.1.1 external NodeManager shuffle service.

Use the Spark 4.1.1 External Shuffle Service

If you require the Spark 4.1.1 external shuffle service, run YARN NodeManager on JDK 17.

After NodeManager runs on JDK 17, it can load spark_shuffle_411.

Changing the Java version used by NodeManager is a cluster-level configuration change. Verify compatibility with other services before you make this change.