Databricks Jobs

Databricks Jobs runs and orchestrates workloads in your Databricks workspace. When you turn on pipeline monitoring for a Databricks data source, ADOC shows each Databricks job as a pipeline, so you can track job runs alongside your other pipelines without instrumenting your code.

ADOC collects run data through an export job that you install in your Databricks workspace. On the schedule that you set in ADOC, the export job sends job and task run data to ADOC. You set this up once per workspace.

Why monitor Databricks jobs in ADOC

• See the run status that Databricks reports for each job run, not a status inferred from the underlying Spark application.
• Check the start time, end time, and status of each run.
• Find the task that caused a job run to fail.
• View the run timeline and lineage on the Pipelines pages, and filter for Databricks pipelines.

How Databricks jobs appear in ADOC

Prerequisites

• An Azure Databricks workspace with Unity Catalog enabled.
• Permission in Databricks to import notebooks into /Workspace/Shared and to create jobs.
• An Azure service principal, with its client ID, client secret, and Azure tenant ID. Pipeline monitoring requires a service principal; personal access tokens aren’t supported. To create one, see Databricks Reliability.
• An ADOC API access key and secret key for the export job. For more information, see API Keys.
• A data plane in the running state.

Step 1: Start adding the Databricks data source in ADOC

1. In the left navigation, click Control Center > Integrations, click Add Data Source, and then select Databricks. To add pipeline monitoring to an existing Databricks data source, edit it instead.
2. On the Name Data Source page, enter a name, select a data plane that's in the running state, and then click Next.
3. On the Connection Details page, set Cloud Provider to AZURE, and then enter the Databricks Workspace Name.
4. Turn on Use Service Principal, and then enter the Databricks URL, Service Principal - Client Id, Service Principal - Client Secret, and AZURE Tenant Id.
5. Enter the JDBC URL, and then set Dataplane Engine to Spark.
6. Under Pipeline monitoring, turn on Enable pipeline monitoring. In Export job name, keep the default name, adoc-pipeline-export, or enter a different name.

When you turn on Enable pipeline monitoring, the Set up the export notebook panel appears on the right. It contains the commands for the next steps, with your ADOC URL and an access token already filled in. Keep this page open while you complete the steps in Databricks.

Step 2: Download the export notebook

In the Set up the export notebook panel, under Download notebook, click Copy. Run the copied command in a terminal to download export_databricks_jobs.py. The command already includes your ADOC URL and an access token, and looks similar to the following:

curl -o export_databricks_jobs.py 'https://<your-adoc-url>/torch/torch-pipeline/api/databricks-jobs/export/notebook' \
  -H 'Authorization: Bearer <access-token>' \
  --compressed

Step 3: Import the notebook into Databricks

1. In your Databricks workspace, go to Workspace > Shared.
2. Click Import, and then select export_databricks_jobs.py.

The notebook path is /Workspace/Shared/export_databricks_jobs.

Step 4: Create the export job

1. In Databricks, create a job with a single notebook task that points to /Workspace/Shared/export_databricks_jobs. You can name the job anything, for example adoc-pipeline-export. Use the same name that you entered in Export job name in Step 1.
2. Add the following notebook parameters:

3. Attach serverless compute or a job cluster to the job.

To create the job from code instead, in the Set up the export notebook panel, under Job definition (YAML), click Copy. Set api_access_key and api_secret_key from a Databricks secret scope, and then deploy the definition with databricks bundle deploy. The definition looks similar to the following:
resources:
  jobs:
    adoc-pipeline-export:
      name: adoc-pipeline-export
      tags:
        acceldata: pipeline-monitoring
      tasks:
        - task_key: export_system_tables
          notebook_task:
            notebook_path: /Workspace/Shared/export_databricks_jobs
            source: WORKSPACE
            base_parameters:
              cp_base_url: "https://<your-adoc-url>"
              api_access_key: "<from-secret-scope>"
              api_secret_key: "<from-secret-scope>"

Step 5: Grant the service principal access to the job

In the job’s permissions, grant your service principal Can Manage Run.

Step 6: Test the connection

1. Return to the Connection Details page in ADOC, and make sure that Export job name matches the name of the export job in Databricks.
2. Click Test Connection, and then click Next.

Step 7: Configure pipeline monitoring

1. On the Set Up Observability page, under Data catalog, turn on Unity Catalog Enabled, and then enter a catalog name in Catalog Name 1. To add more catalogs, click +.
2. Under Pipeline monitoring, configure the following settings:

3. Click Submit.

After you submit, ADOC starts a catalog crawl. If you only need pipeline monitoring, you can cancel the crawl.

View Databricks pipelines

On the Pipelines page, set Pipeline Source to Databricks. To show pipelines from one workspace only, also set Data Source to the name of your Databricks data source. Each Databricks job appears as a pipeline, and the Namespace/Integration column shows the data source name. Runs update on the sync schedule.

To see a pipeline's runs, lineage, and other details, click its name. For more information, see Pipeline Run Details.

Turn off pipeline monitoring

To stop monitoring Databricks jobs, edit the Databricks data source. On the Connection Details page, turn off Enable pipeline monitoring, and then save your changes. ADOC stops discovering new job runs. Existing Databricks pipelines and their runs remain in ADOC until you delete the data source.

Troubleshooting

If Test Connection fails, check the following:

• The service principal client ID, client secret, and tenant ID are correct.
• The job name in ADOC matches the export job name in Databricks exactly.
• Only one job in the workspace has that name.
• The service principal has Can Manage Run on the export job.
• The selected data plane is running.

Known limitations

• Pipeline runs aren’t real time. Runs appear in ADOC after the next scheduled sync.
• Only Azure Databricks is supported.
• Each workspace needs its own export job.