Acceldata
ADOC

Last updated: Nov 14, 2026 05:36 UTC

OpenLineage Integration

ADOC supports OpenLineage as a standard for bringing pipeline observability into your environment without instrumenting your jobs with an SDK. This page explains what OpenLineage is, how ADOC interprets OpenLineage events, and how pipeline and asset metadata is derived from them.

ADOC currently supports the following pipeline orchestrators through OpenLineage:

  • Airflow
  • Spark
  • dbt Core
  • Other OpenLineage-compliant producers. For more information, see Generic OpenLineage producers.

What is OpenLineage

OpenLineage is an open framework for collecting and analyzing data lineage. It defines a standard for recording job execution metadata and specifies three core concepts:

  • Run: A specific instance of a job execution.
  • Job: A process that performs extract, transform, or load (ETL) operations.
  • Dataset: A static data source or destination that a job reads from or writes to.

OpenLineage provides a uniform way to describe these objects and supports adding custom metadata through Facets, which define metadata schemas for a Run, Job, or Dataset. OpenLineage includes a set of default facets, and you can define custom facets as part of your OpenLineage setup.

How ADOC interprets OpenLineage events

ADOC maps each OpenLineage core object to a platform entity:

OpenLineage Core Object

ADOC Platform Entity

Run

Pipeline Run

Job

Pipeline Run Job

Dataset

Asset

The Run and Job objects vary depending on the orchestrator that emits them:

Orchestrator

Run Equivalent

Job Equivalent

Description

Apache Airflow

DAG Run

Task

A DAG Run represents a specific execution of a workflow. A Task is a single unit of work within that workflow.

Apache Spark

Application

Task

An Application represents a Spark job execution. A Task is an individual computation within that application.

DBT Core

Project

Model

A Project refers to a DBT project execution. A Model is a single transformation or SQL operation within that project.

When a run starts in a supported orchestrator — for example, triggering the DAG Finance_HR_Argentina in Airflow — ADOC creates a pipeline and begins tracking it. Subsequent runs of the same DAG are recorded as pipeline runs under the existing pipeline rather than creating new pipelines each time.

Generic OpenLineage producers

In addition to Airflow, Spark, and dbt Core, ADOC accepts OpenLineage run events from any OpenLineage-compliant producer, such as an orchestration tool that ADOC doesn't integrate with natively, an ETL tool, or an in-house application instrumented with an OpenLineage client. ADOC creates pipelines, runs, jobs, datasets, and lineage from the events it receives, so you can observe these pipelines on the same Pipelines page as your other pipelines.

How ADOC processes generic OpenLineage events

  • Event types: ADOC processes run events of type START, COMPLETE, FAIL, and ABORT.
  • Pipelines and lineage: ADOC creates jobs, runs, input and output datasets, and the lineage between them from the events. Column-level lineage is included when the event contains it.
  • Runs without a START event: If a run sends only a COMPLETE event, for example, a backfill or a one-time script, ADOC still creates the run. If a START event arrives later, ADOC corrects the start time of the run.
  • Child events without a parent run: If only task-level events arrive, ADOC groups them under a pipeline named after the parent job in the events.
  • Facets: ADOC processes standard OpenLineage facets. ADOC ignores custom facets that it doesn't support, and these facets don't cause the event to fail.
  • Invalid events: ADOC rejects malformed JSON with an HTTP 400 response and an error message. An invalid event doesn't affect the processing of other valid events.

Set up a producer

Before you begin, make sure that you have the ADOC Control Plane URL for your tenant and an ADOC API key (access key and secret key). For more information, see API Keys.

  1. Configure the OpenLineage HTTP transport of your producer to send run events to ADOC: set the URL to your ADOC Control Plane URL, and set the endpoint to /torch-pipeline/api/v1/lineage.
  2. Configure the transport to authenticate with your ADOC API key. Add the access key and secret key as the accessKey and secretKey request headers.
  3. Run a job, and then verify that the pipeline appears on the Pipelines page.

View generic OpenLineage pipelines

Pipelines created from generic OpenLineage events appear on the Pipelines page with their own source type icon. To show only these pipelines, use the source filter on the Pipelines page. Open a pipeline to view its runs, jobs, datasets, and lineage.

Limitations

  • ADOC processes run events only. ADOC rejects DatasetEvent and JobEvent events as unsupported event types.
  • Datasets that don't match an existing asset still appear in pipeline runs and lineage, but aren't correlated to an asset. For more information, see Asset correlation.

OpenLineage naming convention

In OpenLineage, every entity — jobs, datasets, and so on — is organized within its own namespace. Understanding this convention is useful when filtering or searching for pipelines in ADOC by namespace or job name.

  • Airflow DAG: Uniquely identified as {namespace}.{Airflow DAG Name}
  • Task within a DAG: Identified as {namespace}.{Airflow DAG Name}.{task id}
  • Dataset: Follows source-specific naming conventions. For example, a BigQuery table is identified as bigquery://{project id}.{dataset name}.{table name}

For the full list of naming conventions by source, see OpenLineage naming conventions.

Asset correlation

When an OpenLineage event includes dataset metadata, ADOC checks whether that dataset already exists as an asset in the Data Catalog. If a match is found, ADOC links the pipeline run to the corresponding asset, so the dataset appears in lineage and reliability views alongside the rest of your pipeline.

Asset correlation is currently supported for datasets from the following sources:

  • AWS S3
  • BigQuery
  • GCS
  • Snowflake
  • Redshift
  • Glue
  • Pub/Sub
  • Iceberg
  • PostgreSQL

Datasets from other sources still appear in pipeline runs and lineage graphs, but are not automatically correlated to an existing Catalog asset.

Pipeline tags

When ADOC receives OpenLineage events from Airflow or Spark pipelines, it adds the tags in those events to the matching pipeline. Tags often arrive as single strings, such as Region:US or billing, so ADOC uses the Pipeline Tag Ingestion settings to map each tag to a key and a value.

How tags are mapped

When Tag Key-Value Parsing is on, ADOC maps tags as follows. The examples use the default separator (:) and default key (source_tags).

Source tag

Pipeline tag in ADOC

How it's mapped

Region:US

Region: US

The tag is split into a key and a value at the separator.

env:staging:new

env: staging:new

The tag is split at the first separator only. The rest of the tag, including any later separators, becomes the value.

billing

source_tags: billing

A tag that doesn't contain the separator is added under the default key.

When Tag Key-Value Parsing is off, ADOC uses the whole tag as both the key and the value. For example, Region:US appears as Region:US: Region:US.

Configure tag mapping

The settings apply across your tenant. To change them, you need permission to create or edit pipelines.

  1. In the left navigation, click Settings.
  2. On the ADOC Settings tab, under Pipelines, click Pipeline Tag Ingestion.
  3. Configure the following settings, and then click Save Changes.

Setting

Description

Default

Tag Key-Value Parsing

Turn on to split each tag into a key and a value at the first separator. Turn off to use the whole tag as both the key and the value.

Turned on

Tag Separator

The character that separates the key from the value, for example :, ,, ;, or /. Used only when Tag Key-Value Parsing is on.

:

Tag Default Key

The key for tags that don't contain the separator. Used only when Tag Key-Value Parsing is on.

source_tags

To restore a setting to its default value, click Reset to default next to it, and then click Save Changes.

Changes apply from the next pipeline run. For example, if you change Tag Default Key to app_tags, the next run adds the tag billing as app_tags: billing.

Tag updates on each run

On each run, ADOC updates the pipeline's tags to match the tags in the OpenLineage events. If a tag is removed from the source pipeline, or no longer matches because the settings changed, ADOC removes it from the pipeline. The tag still exists in ADOC, but it's no longer attached to the pipeline.

View, filter, and add tags

To see a pipeline's tags, use the Tags column on the Pipelines page. To filter pipelines by tag, select More Filters > Tags.

You can also attach tags to a pipeline manually. Create the tag in Data Observability Cloud > Governance > Tags, and then edit the pipeline to attach it. For more information, see Tags.

What's next

  • To instrument Airflow or Python pipelines directly with the Acceldata SDK instead of OpenLineage, see Airflow and Python in the Developer Guide.
  • To view and manage pipelines created through OpenLineage, see Working with Pipelines.
  • To understand core pipeline concepts such as runs, spans, and jobs, see Pipeline Observability.
  • To explore column-level lineage across your data assets, see Lineage.