Acceldata
 ADOC
Search documentation… ⌘K

Last updated: Sep 29, 2026 08:50 UTC

OpenLineage for Spark On-Prem or Standalone Clusters

This guide explains how to configure OpenLineage for a Spark application running on a standalone or on-premise Spark cluster, so that ADOC receives lineage events and gives you end-to-end visibility of your Spark pipelines.

Prerequisites

Before you begin, confirm the following:

  • You have a running Spark application on a standalone or on-premise Spark cluster.
  • You have the ADOC Control Plane URL, access key, and secret key for your tenant.
  • Your Spark application’s build system (Gradle, Maven, SBT, or similar) supports adding external dependencies.

Step 1: Add the OpenLineage dependency

OpenLineage requires a Spark agent JAR as a dependency in your Spark application.

Example for Gradle (Kotlin DSL) with a Java Spark application:

implementation("io.openlineage:openlineage-spark_2.12:1.27.0")

Adjust the dependency declaration to match your build system (Maven, SBT, or other).

Step 2: Configure Spark

OpenLineage requires specific Spark configuration values to emit events to the ADOC Control Plane. Add these either through the Spark Session builder’s .config() method, or in the spark-defaults.conf file.

Required configuration values

Configuration key

Value

spark.jars.packages

io.openlineage:openlineage-spark_2.12:1.27.0

spark.extraListeners

io.openlineage.spark.agent.OpenLineageSparkListener

spark.openlineage.transport.url

URL of the ADOC Control Plane

spark.openlineage.transport.type

http

spark.openlineage.transport.endpoint

/torch-pipeline/api/v1/lineage

spark.openlineage.namespace

An appropriate name for your OpenLineage namespace

spark.openlineage.transport.headers.accessKey

Access key generated for your tenant

spark.openlineage.transport.headers.secretKey

Secret key generated for the access key

If keys such as spark.jars.packages or spark.extraListeners already exist in your configuration, append the OpenLineage values as a comma-separated list rather than replacing the existing values.

Example Spark Session builder configuration

SparkSession spark = SparkSession.builder()
.appName("MySparkPipeline")
.config("spark.jars.packages", "io.openlineage:openlineage-spark_2.12:1.27.0")
.config("spark.extraListeners", "io.openlineage.spark.agent.OpenLineageSparkListener")
.config("spark.openlineage.transport.url", "https://<adoc-control-plane-url>")
.config("spark.openlineage.transport.type", "http")
.config("spark.openlineage.transport.endpoint", "/torch-pipeline/api/v1/lineage")
.config("spark.openlineage.namespace", "my-namespace")
.config("spark.openlineage.transport.headers.accessKey", "<ACCESS_KEY>")
.config("spark.openlineage.transport.headers.secretKey", "<SECRET_KEY>")
.getOrCreate();

Step 3: Verify the setup

  • Run your Spark job.
  • Confirm that OpenLineage events arrive at the ADOC Control Plane.
  • Confirm that the Spark application name appears as the pipeline name in the ADOC UI.
  • Open the pipeline in ADOC and confirm that the job and asset nodes appear as expected.

What’s next