YARN Spark Shuffle Services

Update the YARN NodeManager auxiliary service configuration to use the Spark 3.5.5 shuffle service.

1. In the Ambari UI, go to YARN > Configs > Custom yarn-site.

2. Locate the yarn.nodemanager.aux-services property.

3. Replace spark3_shuffle with spark_shuffle_355.

   yarn.nodemanager.aux-services=spark_shuffle_355

4. If Spark 3.3.3 or Spark 3.5.1 services are installed during the upgrade, add the corresponding shuffle services:

   • Spark 3.3.3: spark_shuffle_333

   • Spark 3.5.1: spark_shuffle_351

5. Save the configuration.

6. Restart the affected NodeManagers.