YARN Spark Shuffle Services

Update the YARN NodeManager auxiliary service configuration to use the Spark 3.5.8 shuffle service.

1. In the Ambari UI, go to YARN > Configs > Custom yarn-site.
2. Locate the yarn.nodemanager.aux-services property.
3. Replace spark3_shuffle with spark_shuffle_358.
   

   yarn.nodemanager.aux-services=spark_shuffle_358

4. If Spark 3.3.3, Spark 3.5.1 or Spark 3.5.5 services are installed during the upgrade, add the corresponding shuffle services:
   • Spark 3.3.3: spark_shuffle_333
   • Spark 3.5.1: spark_shuffle_351
   • Spark 3.5.5: spark_shuffle_355
5. Save the configuration.
6. Restart the affected NodeManagers.