OpenLineage Integration

ADOC supports OpenLineage as a standard for bringing pipeline observability into your environment without instrumenting your jobs with an SDK. This page explains what OpenLineage is, how ADOC interprets OpenLineage events, and how pipeline and asset metadata is derived from them.

ADOC currently supports the following pipeline orchestrators through OpenLineage:

• Airflow

• Spark

• DBT Core

What is OpenLineage

OpenLineage is an open framework for collecting and analyzing data lineage. It defines a standard for recording job execution metadata and specifies three core concepts:

• Run: A specific instance of a job execution.

• Job: A process that performs extract, transform, or load (ETL) operations.

• Dataset: A static data source or destination that a job reads from or writes to.

OpenLineage provides a uniform way to describe these objects and supports adding custom metadata through Facets, which define metadata schemas for a Run, Job, or Dataset. OpenLineage includes a set of default facets, and you can define custom facets as part of your OpenLineage setup.

How ADOC interprets OpenLineage events

ADOC maps each OpenLineage core object to a platform entity:

The Run and Job objects vary depending on the orchestrator that emits them:

When a run starts in a supported orchestrator — for example, triggering the DAG Finance_HR_Argentina in Airflow — ADOC creates a pipeline and begins tracking it. Subsequent runs of the same DAG are recorded as pipeline runs under the existing pipeline rather than creating new pipelines each time.

OpenLineage naming convention

In OpenLineage, every entity — jobs, datasets, and so on — is organized within its own namespace. Understanding this convention is useful when filtering or searching for pipelines in ADOC by namespace or job name.

• Airflow DAG: Uniquely identified as {namespace}.{Airflow DAG Name}

• Task within a DAG: Identified as {namespace}.{Airflow DAG Name}.{task id}

• Dataset: Follows source-specific naming conventions. For example, a BigQuery table is identified as bigquery://{project id}.{dataset name}.{table name}

For the full list of naming conventions by source, see OpenLineage naming conventions.

Asset correlation

When an OpenLineage event includes dataset metadata, ADOC checks whether that dataset already exists as an asset in the Data Catalog. If a match is found, ADOC links the pipeline run to the corresponding asset, so the dataset appears in lineage and reliability views alongside the rest of your pipeline.

Asset correlation is currently supported for datasets from the following sources:

• AWS S3

• BigQuery

• GCS

• Snowflake

Datasets from other sources still appear in pipeline runs and lineage graphs, but are not automatically correlated to an existing Catalog asset.

What's next

• To configure OpenLineage with Trino, see Trino Integration with OpenLineage.

• To instrument Airflow or Python pipelines directly with the Acceldata SDK instead of OpenLineage, see Airflow and Python in the Developer Guide.

• To view and manage pipelines created through OpenLineage, see Working with Pipelines.

• To understand core pipeline concepts such as runs, spans, and jobs, see Pipeline Observability.

• To explore column-level lineage across your data assets, see Lineage.