Yarn Optimizer
Objective
Acceldata introduces the Pulse Yarn Optimizer feature, designed to enhance the performance of Yarn Clusters by optimizing the allocation of memory for yarn jobs, leading to a significant performance improvement.
Overview
This feature employs lightweight agents across all Yarn NodeManager nodes within a cluster.
Consider a scenario where a cluster utilizes over 1000 NodeManager nodes to run various jobs like Spark, Hive, and MapReduce. The installation and monitoring of these agents are managed by the Pulse Command Line Interface, known as Accelo. These agents collect real-time metrics from containers scheduled by the Yarn ResourceManager for the job types mentioned. The Pulse Yarn Optimizer then uses these metrics to provide data on available memory, which is not being utilized at all. This influences container scheduling and thereby affects job execution.
The result is a more efficient use of resources, such as memory, allowing for more concurrent jobs and faster execution of subsequent jobs, reducing the need for excessive nodes to manage the workload and minimizing job queuing due to the unavailability of NodeManager slots.
Prerequisites
- A functional cluster with installed Yarn Resource Manager and Node Manager is required.
- The Yarn agents necessitate 10 MB of memory per node for their operation.
- Additionally, the YARN optimizer microservice on the pulse node requires 30MB of free memory to function effectively.
Architecture Design
The YARN Optimizer consists of the following key components:
- Yarn Metrics Agent: Deployed on every YARN node to gather metrics.
- Yarn Resource Optimizer: A Pulse microservice that monitors these metrics and enhances resource utilization across the cluster through the use of specialized algorithms.
Impact on Microservices
The Hadoop Yarn Resource Manager service may experience effects due to this feature, as it directly influences Yarn's scheduling processes.
Risks and Mitigation
The Pulse Yarn Optimizer's primary function is to refine the scheduling decisions of the Yarn Scheduler, a critical component of the Yarn cluster's operations. Under certain conditions, such as abrupt increases in memory demands from multiple concurrent jobs, the adjustments to allow for more concurrent jobs could lead to temporary slowdowns. These delays are often related to garbage collection processes within job runtimes. It's crucial to understand that while some jobs might slow down, job failures are not expected. Should adverse performance impacts be observed, this feature can be temporarily disabled to assess its effects on cluster performance.
Performance Benchmarking
The Pulse team conducted extensive benchmark tests in client environments to evaluate performance enhancements. These tests revealed that many jobs were not fully utilizing available cluster resources. The Pulse Yarn Optimizer detected these inefficiencies, allowing our algorithms to significantly boost resource utilization, as evidenced by the accompanying performance graphs.
Optimization Strategies
Pulse provides the following strategies to optimize the YARN service.
- Buffer Capacity Strategy: This strategy allows overallocation only if any of the yarn nodes is underutilized and any node manager’s available memory is less than the configurable buffer.
- Queue Capacity Strategy: This strategy adds a trigger point to the optimizer. It allows overallocation only if any of the yarn queues is under a heavy load or any node manager’s available memory is less than the configurable buffer.
- Executor Placement Strategy: This strategy distributes the workload across nodes based on the total physical free memory available. This resolves the allocation skew of the executor on the subset in the Yarn scheduler, to depend on the uptime of nodes. For example, the older nodes will not now be allocated with executors first resulting in overcommitment that aggravates this skew.
The Default Capacity Strategy disables the optimization.
If you have the Yarn Optimizer license, Acceldata enables it by default. Then you can proceed with the deployment. Once the deployment is successful, you can see the Yarn Optimizer service appearing on the Pulse UI.
For details about deploying the Yarn Optimizer, see Deploy Yarn Optimizer.
You can navigate to the Pulse UI > Menu and under Yarn Optimizer, you can select the following:
For more details, watch the Yarn Optimizer demo Video.