Databricks Webhook + OpenLineage Setup Runbook

In some Databricks job execution and termination scenarios, the Spark application lifecycle does not reliably invoke the SparkListener.onApplicationEnd() callback that the OpenLineage Spark integration relies on. Because OpenLineage depends on Spark lifecycle callbacks, it may emit a START event without the corresponding terminal COMPLETE or FAIL event, even after the Databricks job has finished.

When the terminal event is missing:

• The lineage run remains stuck in the RUNNING state in ADOC.
• Downstream lineage graph processing and pipeline run status tracking cannot complete correctly.
• Monitoring policies that depend on terminal run events may not evaluate.

This runbook explains how to configure Databricks job webhooks so ADOC can detect the job's terminal state and synthesize the missing terminal lineage event.

Assumptions

• The OpenLineage Spark Databricks integration is already set up and working (transport, listeners, parent lineage, and so on). See OpenLineage for Spark on Databricks.
• You have your ADOC access key and secret key for your tenant.
• You have Databricks workspace admin (or equivalent) permission to create Notification Destinations.

Overview

The Databricks webhook (Notification Destination) is a central, workspace-level configuration. Create the URL and credentials once, under Workspace Settings → Notification Destinations — do not recreate or reconfigure the webhook URL separately for each job.

Creating the destination does not automatically enable it for all jobs. Every applicable execution path must include both the webhook destination ID and the required OpenLineage captured properties:

• For an Airflow-submitted run, pass the destination ID under webhook_notifications in the DAG's Databricks run payload, and add spark.openlineage.capturedProperties to the submitted cluster's spark_conf.
• For a Databricks job cluster, add the destination ID under webhook_notifications in the job definition, and add spark.openlineage.capturedProperties to the job cluster's spark_conf.

The captured properties are required in both cases, because they correlate the OpenLineage START event with the webhook notification.

Step 1: Create the Notification Destination (one-time)

Official Databricks documentation: Create a new notification destination

• Open your Databricks workspace.
• Go to Workspace Settings → Notifications → Notification Destinations.
• Click Add destination.
• Fill in the following fields:

• Replace:
  • <tenant-host> — your ADOC tenant base URL host (for example, your-tenant.acceldata.app)
  • <access_key> / <secret_key> — your ADOC API credentials for that tenant
• Save the destination.

Databricks may send a probe request (type=webhooks.probe) when the destination is created or validated. The ADOC endpoint acknowledges probes with an HTTP 200 response.

Step 2: Get the webhook destination ID

After creating the destination, you need its ID (a UUID) to attach it to jobs.

Option A — Databricks UI:

• Open Workspace Settings → Notifications → Notification Destinations.
• Open the destination you created.
• Copy the destination ID (UUID format, for example 781b5c74-3f61-4669-a3d4-ce31b88f8872).

Option B — Databricks REST API:

curl -s -X GET \ "https://<databricks-workspace-host>/api/2.0/notification-destinations" \ -H "Authorization: Bearer <DATABRICKS_TOKEN>"

Find your destination by name in the response and copy its id.

You'll use this ID in webhook_notifications.on_success and webhook_notifications.on_failure.

Step 3: Attach the webhook to jobs

Configure both on_success and on_failure with the same destination ID, so terminal lineage events are synthesized for both successful and failed runs.

This attachment is required per job or run, even though the Notification Destination itself is created only once.

3.1 Passing the webhook through Airflow DAGs

This assumes other OpenLineage Spark parameters are already configured.

Add webhook_notifications to the Databricks run payload — for example, using DatabricksSubmitRunOperator:

from airflow.providers.databricks.operators.databricks import DatabricksSubmitRunOperatorrun_notebook = DatabricksSubmitRunOperator( task_id='databricks_catalog_task', json={ 'webhook_notifications': { 'on_success': [{'id': '<webhook-destination-id>'}], 'on_failure': [{'id': '<webhook-destination-id>'}], }, }, # ... new_cluster / notebook_task / existing OpenLineage spark_conf ... databricks_conn_id='databricks_default',)

Replace <webhook-destination-id> with the UUID from Step 2.

3.2 Passing the webhook directly on Spark or Databricks jobs

This assumes other OpenLineage Spark parameters are already configured.

In the job definition JSON (Jobs API, Terraform, or UI JSON edit):

{ "webhook_notifications": { "on_success": [ { "id": "<webhook-destination-id>" } ], "on_failure": [ { "id": "<webhook-destination-id>" } ] }}

Step 4: Configure capturedProperties (required)

Until OpenLineage exposes Databricks identifiers on every event, each job must capture the following runtime properties. ADOC uses these values to correlate Databricks webhook notifications with the corresponding OpenLineage runs.

Add spark.openlineage.capturedProperties to the job cluster's Spark configuration. If this setting already exists, append the properties below to the existing comma-separated list instead of replacing it.

Airflow DAG

When the DAG defines the Databricks cluster, add the setting to the cluster's spark_conf:

'spark.openlineage.capturedProperties': ','.join([ 'spark.databricks.clusterUsageTags.clusterOwnerOrgId', 'spark.databricks.clusterUsageTags.clusterName',]),

Databricks job cluster

Add the following key-value pair to the job cluster's Spark configuration:

spark.openlineage.capturedProperties spark.databricks.clusterUsageTags.clusterOwnerOrgId,spark.databricks.clusterUsageTags.clusterName

These properties are used to correlate:

Without capturedProperties, the START event may lack Databricks identifiers, no tracking entry is created, and the webhook cannot synthesize a terminal event.

Configuration examples

Airflow DAG

Minimal pattern showing the webhook, OpenLineage transport, and capturedProperties together. Replace secrets and IDs with your environment's values — do not commit real credentials to source control.

"""Simple DAG to call a Databricks job with OpenLineage + webhook notifications."""from datetime import datetimefrom airflow import DAGfrom airflow.providers.databricks.operators.databricks import DatabricksSubmitRunOperatordefault_args = { 'owner': 'airflow', 'depends_on_past': False, 'email_on_failure': False, 'email_on_retry': False, 'retries': 0,}with DAG( dag_id='databricks_insert_sql_dag', default_args=default_args, description='DAG to call a Databricks job with OpenLineage webhook', schedule_interval=None, start_date=datetime(2026, 1, 11), catchup=False, tags=['databricks', 'openlineage'],) as dag: run_notebook = DatabricksSubmitRunOperator( task_id='databricks_catalog_task', json={ 'webhook_notifications': { 'on_success': [{'id': '<webhook-destination-id>'}], 'on_failure': [{'id': '<webhook-destination-id>'}], }, }, timeout_seconds=3600, new_cluster={ 'spark_version': '17.3.x-scala2.13', 'spark_conf': { 'spark.driver.extraJavaOptions': ( '-Dlog4j2.logger.openlineage.name=io.openlineage ' '-Dlog4j2.logger.openlineage.level=DEBUG' ), 'spark.openlineage.transport.type': 'composite', 'spark.openlineage.transport.continueOnFailure': 'true', 'spark.openlineage.transport.transports.console.type': 'console', 'spark.openlineage.transport.transports.acceldata.type': 'http', 'spark.openlineage.transport.transports.acceldata.url': 'https://<tenant-host>', 'spark.openlineage.transport.transports.acceldata.headers.accessKey': '<access_key>', 'spark.openlineage.transport.transports.acceldata.headers.secretKey': '<secret_key>', 'spark.openlineage.transport.transports.acceldata.endpoint': '/torch-pipeline/api/v1/lineage', 'spark.openlineage.transport.transports.acceldata.timeoutInMillis': '10000', 'spark.openlineage.transport.transports.acceldata.headers.X-User-ID': 'system/databricks', 'spark.openlineage.transport.transports.acceldata.headers.X-User-Name': 'databricks-service', 'spark.openlineage.transport.transports.acceldata.headers.X-Tenant-ID': '<tenant_id>', 'spark.openlineage.transport.transports.acceldata.headers.X-Tenant-Name': '<tenant_name>', 'spark.extraListeners': ( 'io.openlineage.spark.agent.OpenLineageSparkListener,' 'com.databricks.backend.daemon.driver.DBCEventLoggingListener' ), 'spark.executor.extraJavaOptions': '-Dlog4j2.logger.io.openlineage.level=DEBUG', 'spark.openlineage.version': 'v1', 'spark.openlineage.namespace': 'databricks', 'spark.openlineage.capturedProperties': ','.join([ 'spark.databricks.clusterUsageTags.clusterOwnerOrgId', 'spark.databricks.clusterUsageTags.clusterName', ]), 'spark.openlineage.appName': 'databricks_insert_sql_app', # Parent lineage from Airflow (if the OpenLineage Airflow provider is installed) 'spark.openlineage.parentJobNamespace': '{{ macros.OpenLineageProviderPlugin.lineage_job_namespace() }}', 'spark.openlineage.parentJobName': '{{ macros.OpenLineageProviderPlugin.lineage_job_name(task_instance) }}', 'spark.openlineage.parentRunId': '{{ macros.OpenLineageProviderPlugin.lineage_run_id(task_instance) }}', 'spark.openlineage.rootParentJobNamespace': '{{ macros.OpenLineageProviderPlugin.lineage_job_namespace() }}', 'spark.openlineage.rootParentJobName': '{{ macros.OpenLineageProviderPlugin.lineage_job_name(task_instance) }}', 'spark.openlineage.rootParentRunId': '{{ macros.OpenLineageProviderPlugin.lineage_run_id(task_instance) }}', 'spark.databricks.job.name': '{{ task.task_id }}', }, 'node_type_id': 'r6id.xlarge', 'num_workers': 1, # ... remaining cluster settings (tags, init scripts, security mode, etc.) ... }, notebook_task={ 'notebook_path': '/Workspace/Users/<user>/your_notebook', }, databricks_conn_id='databricks_default', ) run_notebook

Databricks job cluster (YAML)

The following Databricks Job YAML example configures webhook notifications and the required OpenLineage settings for a shared job cluster:

resources: jobs: databricks_webhook_job: name: Databricks webhook job webhook_notifications: on_success: - id: <webhook-destination-id> on_failure: - id: <webhook-destination-id> tasks: - task_key: databricks_spark_app_webhook_test_job notebook_task: notebook_path: /Workspace/Users/<user>/insert_job.py source: WORKSPACE job_cluster_key: job_cluster job_clusters: - job_cluster_key: job_cluster new_cluster: cluster_name: "" spark_version: 17.3.x-scala2.13 node_type_id: m6g.xlarge data_security_mode: DATA_SECURITY_MODE_AUTO runtime_engine: STANDARD kind: CLASSIC_PREVIEW is_single_node: false enable_elastic_disk: true autoscale: min_workers: 2 max_workers: 8 aws_attributes: first_on_demand: 1 availability: SPOT_WITH_FALLBACK zone_id: auto spot_bid_price_percent: 100 custom_tags: owner: <owner> purpose: <purpose> spark_env_vars: PYSPARK_PYTHON: /databricks/python3/bin/python3 spark_conf: spark.app.name: insert_data_job spark.extraListeners: io.openlineage.spark.agent.OpenLineageSparkListener spark.openlineage.transport.type: http spark.openlineage.transport.url: https://<tenant-host> spark.openlineage.transport.endpoint: /torch-pipeline/api/v1/lineage spark.openlineage.transport.headers.accessKey: <access-key> spark.openlineage.transport.headers.secretKey: <secret-key> spark.openlineage.namespace: <openlineage-namespace> spark.openlineage.version: v1 spark.openlineage.client.logging.level: DEBUG spark.openlineage.capturedProperties: >- spark.databricks.clusterUsageTags.clusterOwnerOrgId, spark.databricks.clusterUsageTags.clusterName spark.openlineage.facets.spark.logicalPlan.enabled: "true" spark.openlineage.facets.spark_unknown.enabled: "true" spark.openlineage.facets.debug.enabled: "true" init_scripts: - volumes: destination: /Volumes/<catalog>/<schema>/<volume>/init-script.sh queue: enabled: true

Replace all placeholder values with settings for your environment. Store your ADOC credentials in a secrets manager, or inject them during deployment, instead of committing plaintext keys.

Verification checklist

After setup, confirm:

• The Notification Destination exists and the probe or auth check succeeds (no persistent 401 errors).
• The job JSON includes webhook_notifications for both on_success and on_failure.
• The Spark conf includes spark.openlineage.capturedProperties with the Databricks properties listed above.
• After running a job:
  • The OpenLineage START event arrives in ADOC.
  • On job completion, Databricks fires the webhook.
  • The pipeline run moves to COMPLETED or FAILED, and is not stuck in RUNNING.

Troubleshooting

Quick reference

Webhook URL: https://<tenant>/torch-pipeline/api/v1/openlineage/databricks/webhookAuth (Notification Destination): User Name = access_key Password = secret_keyJob attachment: webhook_notifications.on_success = [{ "id": "<destination-id>" }] webhook_notifications.on_failure = [{ "id": "<destination-id>" }]Required Spark conf: spark.openlineage.capturedProperties = spark.databricks.clusterUsageTags.clusterOwnerOrgId,spark.databricks.clusterUsageTags.clusterName

What's next

• For the base Databricks Spark OpenLineage setup this runbook extends, see OpenLineage for Spark on Databricks.