Resource Optimization for YARN Static Queues

This use case demonstrates how Pulse YARN Optimizer optimizes resource allocation in static queues through a practical example.

Pulse YARN Optimizer employs lightweight Agents across all YARN NodeManager nodes within a cluster. These agents collect real-time metrics from containers scheduled by the Yarn ResourceManager for the job types mentioned. The Pulse Yarn Optimizer then uses these metrics to provide data on available memory that is not being utilized at all. This influences container scheduling and thereby affects job execution.

Pulse allows you to configure the Optimization Ratio in Pulse YARN Optimizer > Config > Static Queues. In this section, you can select a queue from the drop-down menu and set the desired ratio.

In this example use case, let’s take if YARN has a total of 100 GB of resources distributed across three queues as configured in Ambari:

• Q1: 50% (maximum allowed)
• Q2: 25%
• Q3: 25%

If Pulse YARN Optimizer inflates the total resource availability to 110 GB, adding 10 GB of extra capacity, this extra memory is distributed among the queues based on their configured percentages.

Optimized Distribution:

• Q1 (50%) → gets 5 GB

• Q2 (25%) → gets 2.5 GB

  • If Q2 has sub-queues with absolute resource configurations (static queues):

    • Q21: If you set the Optimization Ratio to .7 in Pulse YARN Optimizer > Config > Static Queue, the resources optimized will be → 1.75 GB (2.5 GB × 0.7).
    • Q22: If you set the Optimization Ratio to .3 in Pulse YARN Optimizer > Config > Static Queue, the resources optimized will be → 0.75 GB (2.5 × 0.3).

• Q3 (25%) → gets 2.5 GB:

  • If Q3 has sub-queues with absolute resource configurations (static queues):
    • Q 31: If you set the Optimization Ratio to 0.6, the resources optimized will be → 1.5 GB (2.5 GB × 0.6).
    • Q 32: If you set the Optimization Ratio to 0.4, the resources optimized will be → 1 GB (2.5 GB × 0.4).