OpenLineage for Spark on Databricks

Databricks provides a managed platform for running Spark jobs, AI/ML workflows, and cataloging. This guide explains how to configure OpenLineage for Spark jobs on Databricks job clusters so that lineage events reach the ADOC Control Plane.

This procedure applies to Databricks Runtime on both AWS and Azure.

Databricks cluster types

Databricks includes a built-in Spark session on every cluster. When the cluster starts, it initiates the Spark session, which triggers the Spark application start OpenLineage event. Cluster type affects when that event fires:

• All-purpose clusters: Stay active and shut down only after a period of inactivity; they are never deleted outright. If the cluster is already running when a job is scheduled, the job reuses the existing Spark session, so OpenLineage does not send a new Spark application start event.

• Job clusters: Created when a job is scheduled and deleted when the job completes. Each run starts a new Spark session, generating both the Spark application START and COMPLETE OpenLineage events.

OpenLineage support is limited to job clusters. All-purpose clusters are not supported, due to this Spark session behavior.

Prerequisites

• Databricks CLI installed and authenticated on your local machine.

• The OpenLineage Spark JAR downloaded (for example, openlineage-spark_2.12-1.28.0.jar).

• The ADOC Control Plane URL, access key, and secret key for your tenant.

Step 1: Upload the OpenLineage JAR to DBFS

Create a shell script to upload the OpenLineage JAR to DBFS:

This example uses /databricks/openlineage as the DBFS location; you can choose any suitable path. Ensure you are authenticated to the Databricks workspace you want to run jobs on and observe with ADOC before running the script. For details on configuring and authenticating with the Databricks CLI, refer to this doc.

Step 2: Create an init script for job clusters

Create an init script to configure OpenLineage at cluster start-up, then upload it to DBFS. The job cluster configuration will reference this script.

Step 3: Trigger Spark jobs in Databricks

Spark jobs on Databricks run in one of two ways: through Databricks Workflows, or through Databricks Notebooks. When triggering a Workflow, configure a job cluster where the Workflow will run, using a Spark version compatible with the OpenLineage Spark JAR you downloaded.

Spark configuration

Add the following in Advanced Options → Spark → Spark Config:

Init scripts

In Advanced Options → Spark → Init Scripts, add the DBFS path to the init script you uploaded in Step 2.

With this configuration, the Workflow sends OpenLineage events to the ADOC Control Plane whenever it runs. The same configuration also supports triggering Databricks Notebook execution.

Step 4: Configure notebook-specific behavior

To emit the SparkApplicationEnd (COMPLETE) event for notebooks:

1. Avoid calling spark.stop() directly — it can cause notebooks to fail or run indefinitely.

2. Add the following code as the last step of the notebook, to trigger the OpenLineage listener directly:

Why this is necessary: With job clusters, the Spark application COMPLETE event is not always emitted when the cluster terminates. Adding an explicit spark.stop() at the end of the last notebook in a workflow can cause the job to be marked FAILED in Databricks even though it completed.

Without either spark.stop() or the workaround above, pipelines in ADOC can remain in the RUNNING state indefinitely. Adding the script above as the last notebook step emits the COMPLETE event directly to the OpenLineage listener, without triggering the spark.stop() failure behavior.

Trigger Databricks jobs from Airflow

A common pattern is an Airflow DAG that triggers a Databricks Spark job. Even though these are separate pipelines, ADOC can visualize the relationship between them.

With the Spark OpenLineage integration, ADOC’s inter-pipeline linkage mechanism displays relationships between pipelines across different integrations — including an Airflow DAG that triggers a Databricks job.

Example: DatabricksSubmitRunOperator

Parent metadata configuration

The following three Spark configuration values establish the linkage between the Airflow task and the Databricks pipeline:

These inject the Airflow task details into the Spark job configuration. When the Spark job emits its START event, ADOC receives the parent job information, which enables:

• Automatic visualization of the Airflow-to-Databricks pipeline linkage.

• Clickable links in the ADOC UI from the Airflow task to the Databricks pipeline.

Caveats

1. The example above demonstrates triggering a notebook.

2. For Databricks Workflows, use DatabricksRunNowOperator. Because the cluster configuration is pre-defined for Workflows, dynamic parent linkage cannot be injected the same way.

3. To maintain Airflow-to-Databricks lineage for Workflows, the parent metadata (parentJobNamespace, parentJobName, parentRunId) must be passed dynamically at runtime. This is an area of active development; check back for updates on Workflow support.

What’s next

• If jobs occasionally remain stuck in a RUNNING state due to a missing terminal event, see Databricks webhook + OpenLineage setup runbook.

• To confirm your datasets link correctly to Catalog assets, see Asset correlation for Redshift, Glue, Pub/Sub, and Iceberg.