Airflow OpenLineage Integration

This page describes how to set up OpenLineage in an Airflow environment so ADOC can observe DAG runs without requiring any code instrumentation.

Why use OpenLineage for Airflow

ADOC previously supported Airflow DAG observability through two approaches, both now deprecated:

Acceldata Airflow SDK (deprecated): Installed as an Airflow plugin. Provided moderate to full instrumentation of DAG code, and allowed DAGs to be viewed in Airflow itself.
Acceldata Listener Plugin (deprecated): A no-code approach for visualizing DAGs in ADOC. Extended the Airflow Listener interface and called the ADOC API based on events the listener captured.

Both approaches had limitations that OpenLineage resolves:

No asset linking: During DAG execution, tasks often interact with data assets, but the deprecated approaches could not detect them. OpenLineage solves this by correlating asset metadata from DAG runs with assets in the Data Catalog, enabling automatic data lineage.

Limited task detection: Conditional executions, such as SKIPPED tasks, were not detected, and tasks like EmptyOperator or DummyOperator were invisible. OpenLineage captures these tasks as well, giving you complete lineage visibility.

OpenLineage is now the supported path for Airflow observability in ADOC. If you're still using the Acceldata Airflow SDK or Listener Plugin, plan to migrate to OpenLineage using the setup guides below.

Setting up the integration

Configuration depends on where your Airflow environment runs. See the guide for your environment:

What's next

If your Airflow DAGs trigger dbt Cloud jobs, see Link Airflow tasks to dbt Cloud pipeline runs in ADOC to associate them correctly.
For an overview of how ADOC interprets OpenLineage events generally, see OpenLineage Integration.

Last updated on

Was this page helpful?