OpenLineage for Spark On-Prem or Standalone Clusters

This guide explains how to configure OpenLineage for a Spark application running on a standalone or on-premise Spark cluster, so that ADOC receives lineage events and gives you end-to-end visibility of your Spark pipelines.

Prerequisites

Before you begin, confirm the following:

You have a running Spark application on a standalone or on-premise Spark cluster.
You have the ADOC Control Plane URL, access key, and secret key for your tenant.
Your Spark application’s build system (Gradle, Maven, SBT, or similar) supports adding external dependencies.

Step 1: Add the OpenLineage dependency

OpenLineage requires a Spark agent JAR as a dependency in your Spark application.

Example for Gradle (Kotlin DSL) with a Java Spark application:

implementation("io.openlineage:openlineage-spark_2.12:1.27.0")

Info Adjust the dependency declaration to match your build system (Maven, SBT, or other).

Step 2: Configure Spark

OpenLineage requires specific Spark configuration values to emit events to the ADOC Control Plane. Add these either through the Spark Session builder’s .config() method, or in the spark-defaults.conf file.

Required configuration values

Configuration key	Value
spark.jars.packages	io.openlineage:openlineage-spark_2.12:1.27.0
spark.extraListeners	io.openlineage.spark.agent.OpenLineageSparkListener
spark.openlineage.transport.url	URL of the ADOC Control Plane
spark.openlineage.transport.type	http
spark.openlineage.transport.endpoint	/torch-pipeline/api/v1/lineage
spark.openlineage.namespace	An appropriate name for your OpenLineage namespace
spark.openlineage.transport.headers.accessKey	Access key generated for your tenant
spark.openlineage.transport.headers.secretKey	Secret key generated for the access key

Info If keys such as spark.jars.packages or spark.extraListeners already exist in your configuration, append the OpenLineage values as a comma-separated list rather than replacing the existing values.

Example Spark Session builder configuration

SparkSession spark = SparkSession.builder() .appName("MySparkPipeline") .config("spark.jars.packages", "io.openlineage:openlineage-spark_2.12:1.27.0") .config("spark.extraListeners", "io.openlineage.spark.agent.OpenLineageSparkListener") .config("spark.openlineage.transport.url", "https://<adoc-control-plane-url>") .config("spark.openlineage.transport.type", "http") .config("spark.openlineage.transport.endpoint", "/torch-pipeline/api/v1/lineage") .config("spark.openlineage.namespace", "my-namespace") .config("spark.openlineage.transport.headers.accessKey", "<ACCESS_KEY>") .config("spark.openlineage.transport.headers.secretKey", "<SECRET_KEY>") .getOrCreate();

Step 3: Verify the setup

Run your Spark job.
Confirm that OpenLineage events arrive at the ADOC Control Plane.
Confirm that the Spark application name appears as the pipeline name in the ADOC UI.
Open the pipeline in ADOC and confirm that the job and asset nodes appear as expected.

What’s next

To confirm your datasets link correctly to Catalog assets, see Asset correlation for Redshift, Glue, Pub/Sub, and Iceberg.
For an overview of how ADOC interprets Spark OpenLineage events, see Spark OpenLineage Integration.

Last updated on

Was this page helpful?