Airflow Dashboard

The Airflow Dashboard provides a high-level operational summary of the Airflow cluster.

Steps

1. In the Pulse UI, go to Airflow on the left pane.
2. Select Dashboard.

K8s Metrics

• Avg Task Adoption Duration: Average time taken by the executor to adopt orphaned tasks.
• Avg Clear Queued Task Duration: Average time taken to clear queued tasks.

Operator Successes by Type

• Bash Operator: Count of successful BashOperator executions.
• Python Operator: Count of successful PythonOperator executions.
• Branch Python Operator: Count of successful BranchPythonOperator executions.

Airflow Executor Summary

• Total DAGs: Total number of DAGs available in the environment.
• Executor Open Slots: Number of available task execution slots.
• Executor Queued Tasks: Tasks waiting to be executed.
• Executor Running Tasks: Tasks currently running.
• Avg DAG Queue Time:* Average time DAG runs spend in the queue before execution.

Triggerer Heartbeat

• Shows heartbeat signals from the Airflow Triggerer to confirm it is alive and responsive.

Scheduler Heartbeat Trend

• Displays scheduler heartbeat over time to detect scheduler lag or failures.

Airflow DAG Run Stats

• Avg DAG Dependency Check: Average time spent evaluating DAG dependencies.
• Avg DAG Success State: Average time taken for successful DAG completion.
• Avg DAG Failed State: Average time taken for failed DAG runs.
• Avg DAG Schedule Delay: Delay between scheduled time and actual execution start.

Airflow DAG Processing Stats

• Avg Parse Time: Time taken to parse DAG files.
• Files to be Scanned: Number of DAG files pending parsing.
• Filesystem Updates: Number of detected file updates.
• Parse Errors: Number of DAG parsing errors.

DAG Run Types

Displays distribution of:

• Scheduled runs
• Manual runs

DAG Run States

Distribution of:

• Success
• Failed
• Running
• Other states

Task States

Distribution of:

• Success
• Running
• Failed
• Skipped

Airflow Scheduler Stats

• Time Spent on Critical Section: The time the scheduler spends in the critical execution block.
• Orphaned Tasks Adopted: Count of orphaned tasks adopted.
• Orphaned Tasks Cleared: Count of cleared orphaned tasks.
• Loop Duration: Scheduler loop execution duration.
• Task Executable: Tasks eligible for execution.
• Task Starving: Tasks waiting too long for resources.

Charts

• Operator Success Over Time: Trend of successful operator executions across time.

• Task Instances Created: Number of task instances created over time.

• DAG Duration Trend: Trend showing elapsed DAG execution time across runs.

• DAG File Load Times: Table showing DAG load performance:

  • DAG ID
  • Load time

• Pool Utilization Trend — Running: Tracks running tasks consuming pool slots.

• Pool Utilization Trend — Queued: Tracks queued tasks waiting for pool slots.

• Executor Load Over Time — Queued: Number of tasks queued in the executor over time.

• Executor Load Over Time — Running: Number of tasks running in the executor over time.