Spark OpenLineage Integration

ADOC uses OpenLineage to automatically capture lineage from Apache Spark applications, including the jobs and datasets each application interacts with. This page explains how ADOC interprets Spark OpenLineage events and links to the setup guide for your environment.

Apache Spark overview

Apache Spark is a distributed, multi-language engine used for data engineering, data science, and machine learning. Each executed workload is a Spark application, which Spark internally breaks into tasks based on its logical execution plan.

ADOC maps these Spark concepts to platform entities as follows:

Spark concept	ADOC entity
Spark application	Pipeline
Task within a Spark application	Job node
Dataset read from or written to	Asset node

What the integration adds

Before the OpenLineage integration, ADOC had no visibility into Spark data lineage. With OpenLineage configured:

Events emitted during Spark execution are processed automatically — no additional code instrumentation is required.
Job and dataset lineage are inferred automatically from the events.
Both Databricks and on-premise or standalone Spark applications are supported.

Supported capabilities

Capability	Supported
Automated job lineage	Yes
Reliability and monitoring policies	Yes
Automated table asset lineage	Yes
Automated column-level asset lineage	No

Supported engines

Databricks
Apache Spark (on-premise or standalone)

Setting up the integration

Configuration depends on where your Spark application runs. See the guide that matches your environment:

What’s next

To confirm your datasets link correctly to Catalog assets after setup, see Asset correlation for Redshift, Glue, Pub/Sub, and Iceberg.
To understand how ADOC interprets OpenLineage events generally, see ADOC OpenLineage Integration.

Last updated on

Was this page helpful?