Airflow Dashboard

The Airflow Dashboard provides a centralized view of workflow execution, scheduler health, executor performance, DAG processing, and task activity.

Use this dashboard to monitor Airflow operations, track workflow execution trends, identify scheduling bottlenecks, and ensure the health of your orchestration environment.

Navigation Steps

1. In the xObserve UI, navigate to Airflow from the left navigation pane.
2. Select Airflow > Dashboard.
3. Select the desired cluster from the cluster selector in the upper-right corner.

The Airflow Dashboard opens, displaying workflow, scheduler, executor, and task metrics.

Time Period

• Use the Time Period selector in the upper-left corner of the dashboard to filter metrics and charts for a specific monitoring window.

Step Interval

• This controls how often the dashboard needs to refresh.

Metrics

Kubernetes Metrics

Monitor Kubernetes-related task scheduling operations.

• Avg Task Adoption Duration – Average time required for Airflow to adopt orphaned tasks.
• Avg Clear Queued Task Duration – Average time required to clear queued tasks.

Airflow Executor Summary

Monitor executor capacity and task processing status.

• Total DAGs – Total number of DAGs managed by Airflow.
• Executor Open Slots – Number of executor slots available for task execution.
• Executor Queued Tasks – Number of tasks waiting for execution.
• Executor Running Tasks – Number of tasks currently being executed.
• Avg DAG Queue Time – Average time tasks spend waiting in the execution queue.

Scheduler and Triggerer Health

Monitor scheduler and triggerer availability and responsiveness.

• Triggerer Heartbeat – Indicates the health and responsiveness of the Airflow Triggerer.
• Scheduler Heartbeat Trend – Tracks scheduler heartbeat activity over time.

DAG Processing Metrics

Monitor DAG parsing and processing performance.

• Avg Parse Time – Average time required to parse DAG files.
• Files to be Scanned – Number of DAG files pending scan.
• Filesystem Updates – Number of detected DAG file updates.
• Parse Errors – Number of DAG parsing errors.

DAG Run Metrics

Monitor DAG execution performance.

• Avg DAG Dependency Check – Average time spent validating DAG dependencies.
• Avg DAG Success State – Average time for DAG runs to reach a successful state.
• Avg DAG Failed State – Average time for DAG runs to reach a failed state.
• Avg DAG Schedule Delay – Average delay between scheduled and actual DAG execution.

Airflow Scheduler Statistics

Monitor scheduler efficiency and workload management.

• Time Spent on Critical Section – Time spent by the scheduler in critical scheduling operations.
• Orphaned Tasks Adopted – Number of orphaned tasks successfully adopted.
• Orphaned Tasks Cleared – Number of orphaned tasks removed by the scheduler.
• Loop Duration – Time required to complete a scheduler loop.
• Task Executable – Number of tasks eligible for execution.
• Task Starving – Number of tasks unable to obtain execution resources.

Charts

DAG and Task Distribution

• DAG Run Types – Distribution of DAG run types.
• DAG Run States – Distribution of DAG execution states.
• Task States – Distribution of task execution states.

Workflow Trends

• Operator Successes Over Time – Number of successful operator executions over time.
• Task Instances Created – Number of task instances created over time.
• DAG Duration Trend – Execution duration trends for DAG runs.
• DAG File Load Times – Time required to load and process DAG files.

Resource Utilization

• Pool Utilization Trend – Running – Running tasks utilizing Airflow pools.
• Pool Utilization Trend – Queued – Queued tasks waiting for pool resources.
• Executor Load Over Time – Running – Executor load generated by running tasks.
• Executor Load Over Time – Queued – Executor load generated by queued tasks.