Visualize YARN Resource Metrics
The YARN Optimizer Dashboard provides a comprehensive view of resource utilization before and after optimization. It helps you monitor cluster performance, track resource allocation improvements, and identify potential areas for further tuning.
You can use the dashboards to understand how the YARN Optimizer improves memory, CPU, and container efficiency across the cluster.
Accessing the Dashboard
- In the Pulse UI, select YARN Optimizer from the left navigation bar.
- On the Dashboard page, set the time range and step interval to update the charts and display the latest data.
Summary Panel
The Summary Panel gives a quick comparison of cluster resources before and after optimization.
It highlights improvements in memory and core usage, along with container scaling details.
Displayed Metrics:
- Original Memory: Total memory assigned to YARN
- Original Cores: Total vCores assigned to YARN
- Optimized Memory: Inflated or overcommitted memory beyond physically available memory after optimization
- Optimized Cores: Updated core allocation after optimization
- Containers Increased:* Number of additional containers allocated due to optimization
The Original Memory and vCores (YARN) values shown in the Summary Panel include data only from nodes where the YARN Optimizer Agent is enabled. Nodes without the agent enabled are excluded from these calculations.
Use this panel to assess the impact of optimization at a glance.
Charts
The Charts section provides detailed visualizations of how YARN resources are allocated, utilized, and optimized over time.
The Original Memory and vCores (YARN) values considered for calculation in these charts represent the totals for all nodes in the cluster, regardless of whether the YARN Optimizer Agent is enabled on those nodes.
Total YARN Memory
- Shows the total available YARN memory across the cluster.
- Calculation: Total YARN Memory = Original Memory + Optimized Memory
This helps you visualize memory scaling and utilization trends after optimization.
Total YARN CPU
- Displays the total number of YARN vCores in use.
- Calculation: Total YARN vCores = Original vCores + Optimized vCores
This chart shows how CPU capacity improves over time through optimization.
YARN Memory and CPU Overview
This chart provides a detailed visualization of how memory and CPU resources are distributed and allocated across the cluster.
The Original Memory and vCores (YARN) values shown in these charts include data only from nodes where the YARN Optimizer Agent is enabled. Nodes without the agent enabled are excluded from these calculations.
Memory Metrics
The Memory view shows how memory is utilized before and after optimization. It helps compare total capacity, YARN allocation, and actual usage to evaluate optimization efficiency.
Displayed Metrics:
| Chart | Description |
|---|---|
| Physical Memory | Total physical (hardware) memory available across all nodes. Example: 5 nodes × 100 GB each = 500 GB total. |
| Original YARN Memory | Total memory assigned to YARN before optimization. Example: 80 GB per node × 5 nodes = 400 GB. |
| Allocated Memory | Portion of YARN memory assigned to running containers or workloads. Example: 5 containers × 1 GB = 5 GB allocated. |
| Memory Used | Actual memory consumed by active containers. Example: 5 containers use 200 MB each → 1 GB total used. |
| New Memory (Total Memory): | Combined total of Original YARN Memory and additional optimized (overcommitted) memory available after optimization. |
This view helps identify how much of the allocated YARN memory is truly utilized and how optimization increases the effective memory pool through overcommitment.
CPU Metrics
To view CPU resource distribution, click the CPU tab on the right side of the chart.
Displayed Metrics:
| Chart | Description |
|---|---|
| Physical CPU Cores | Total number of physical CPU cores available across all nodes. Example: 5 nodes × 16 cores each = 80 cores total. |
| Original Cores | Total CPU cores allocated to YARN before optimization. Example: 12 cores per node × 5 nodes = 60 cores. |
| Allocated Cores | Number of CPU cores currently assigned by YARN to active jobs or containers. Example: 5 running containers × 1 core = 5 cores allocated. |
| CPU Used | Actual CPU utilization by running workloads in real time. Example: Out of 5 allocated cores, only 2 cores are actively used = 40% CPU usage. |
| New Cores (Total Cores) | Combined total of Original Cores and any optimized (overcommitted) cores available after optimization. |
This view helps identify how much of the allocated YARN memory is truly utilized and how optimization increases the effective memory pool through overcommitment.
Cluster Application Details
Shows the overall health and state of cluster workloads over time.
Categories Displayed:
- Running
- Pending
- Completed
- Failed
- Killed
This view helps you monitor workload stability and performance trends post-optimization.
Memory Utilization by Number of Hosts
Displays memory usage across all hosts in the cluster.
- Click any section of the pie chart to view detailed memory metrics for individual hosts.
- The visualization shows an increase in CPU utilization, memory usage, and container count after optimization.
- Click Expand › to view detailed node-level resource usage.
CPU Utilization by Number of Hosts
Displays CPU utilization across all hosts in the cluster.
- Click any section of the pie chart to view per-host CPU usage details.
- The visualization shows an increase in CPU utilization, memory usage, and container count after optimization.
- Click Expand › to view detailed node-level resource usage.
Summary
The YARN Optimizer Dashboard provides a unified, visual view of how cluster resources are utilized and optimized.