Yarn Optimizer Dashboard
After enabling the Yarn Optimizer on Pulse, it provides the overview of Yarn Clusters within your ecosystem. You can monitor the optimized resource details, including the cluster memory usage, cluster nodes optimized, etc.
You can navigate to the Pulse UI > Menu and select Yarn Optimizer > Dashboard.
Cluster Memory Usage
This chart provides the following details:
- Total Memory: The total amount of memory available in a cluster.
- Used Memory: The average amount of memory utilized (in GBs) in a cluster.
This graph helps you monitor, optimize, and plan memory resource allocation within a cluster, ensuring that memory usage is efficiently aligned with available resources.
Cluster Nodes Optimized
This chart provides the following details:
- Total: The total number of nodes available in a cluster.
- Optimized: The total number of optimized nodes in a cluster.
This graph displays the number of nodes available in a cluster, and when the Optimization is enabled, the optimized nodes with details are shown on the graph.
You can configure the number of nodes in a cluster that needs to be optimized. For details, see Yarn Optimizer Configuration.
On the Cluster Node Optimized graph, you can click on a point where the nodes are optimized and check for the following details:
- Host: The list of nodes in a cluster.
- Optimization Applied: Shows whether the optimization was applied or not by showing Yes or No for each node.
- Optimization Strategy: The Optimization Strategy applied is either Buffer Capacity Strategy, Queue Capacity Strategy, or Executor Placement Strategy.
Agent Status
This chart provides the following details:
- Total: The total number of Agents installed in a cluster.
- Down: The status of the Agent on each node—Up or Down.
- Yarn Node Manager: The count of Yarn Node Managers in a cluster.
On every node, the Yarn Metrics Agent is installed to collect the metrics. This graph shows the total number of installed Agents, the number of Down Agents, and the count of Yarn Node Managers in the cluster.
Yarn Allocated vs. Overcommit Memory
This chart provides the following details:
- Allocated Memory: Presents the total allocated memory over time in GBs for Yarn jobs.
- OverCommit Memory: Displays the amount of optimized memory usage in GBs within the Yarn cluster over time for various strategies.
This graph shows how much the cluster is optimized.
For example, If a job is allocated with 325 GB of memory, the YARN Optimizer continuously monitors the actual container usage and visually represents the unused portion of the allocated memory through a graph. In this example, 140 GB of memory remained unused and was efficiently optimized by the YARN Optimizer, ensuring better resource utilization.
The Heat Map of Optimized Node Memory
This chart shows the percentage of optimized nodes summary over a time period.
