Airflow Dashboard

The Airflow Dashboard provides a high-level operational summary of the Airflow cluster.

Steps

1. In the Pulse UI, go to Airflow on the left pane.

2. Select Dashboard.

K8s Metrics

• Avg Task Adoption Duration: Average time taken by the executor to adopt orphaned tasks.

• Avg Clear Queued Task Duration: Average time taken to clear queued tasks.

Operator Successes by Type

• Bash Operator: Count of successful BashOperator executions.

• Python Operator: Count of successful PythonOperator executions.

• Branch Python Operator: Count of successful BranchPythonOperator executions.

Airflow Executor Summary

• Total DAGs: Total number of DAGs available in the environment.

• Executor Open Slots: Number of available task execution slots.

• Executor Queued Tasks: Tasks waiting to be executed.

• Executor Running Tasks: Tasks currently running.

• Avg DAG Queue Time:* Average time DAG runs spend in the queue before execution.

Triggerer Heartbeat

• Shows heartbeat signals from the Airflow Triggerer to confirm it is alive and responsive.

Scheduler Heartbeat Trend

• Displays scheduler heartbeat over time to detect scheduler lag or failures.

Airflow DAG Run Stats

• Avg DAG Dependency Check: Average time spent evaluating DAG dependencies.

• Avg DAG Success State: Average time taken for successful DAG completion.

• Avg DAG Failed State: Average time taken for failed DAG runs.

• Avg DAG Schedule Delay: Delay between scheduled time and actual execution start.

Airflow DAG Processing Stats

• Avg Parse Time: Time taken to parse DAG files.

• Files to be Scanned: Number of DAG files pending parsing.

• Filesystem Updates: Number of detected file updates.

• Parse Errors: Number of DAG parsing errors.

DAG Run Types

Displays distribution of:

• Scheduled runs

• Manual runs

DAG Run States

Distribution of:

• Success

• Failed

• Running

• Other states

Task States

Distribution of:

• Success

• Running

• Failed

• Skipped

Airflow Scheduler Stats

• Time Spent on Critical Section: The time the scheduler spends in the critical execution block.

• Orphaned Tasks Adopted: Count of orphaned tasks adopted.

• Orphaned Tasks Cleared: Count of cleared orphaned tasks.

• Loop Duration: Scheduler loop execution duration.

• Task Executable: Tasks eligible for execution.

• Task Starving: Tasks waiting too long for resources.

Charts

• Operator Success Over Time: Trend of successful operator executions across time.

• Task Instances Created: Number of task instances created over time.

• DAG Duration Trend: Trend showing elapsed DAG execution time across runs.

• DAG File Load Times: Table showing DAG load performance:

  • DAG ID

  • Load time

• Pool Utilization Trend — Running: Tracks running tasks consuming pool slots.

• Pool Utilization Trend — Queued: Tracks queued tasks waiting for pool slots.

• Executor Load Over Time — Queued: Number of tasks queued in the executor over time.

• Executor Load Over Time — Running: Number of tasks running in the executor over time.