Working with Pipelines

Pipelines in ADOC can be created and managed in different ways, depending on your workflow. You do not always need to create them manually. Pipelines can also be registered automatically when metadata is reported through supported integrations.

Create a Pipeline

Pipelines can appear in ADOC in two ways: automatic creation and manual creation.

Automatic Creation (Preferred)

Pipelines can be created automatically when metadata is reported through supported connectors or through OpenLineage.

Manual Creation (UI or SDK)

Use manual creation when you want to model a workflow before instrumenting jobs, assign ownership, teams, or tags up front, or define the logical structure of jobs and assets.

To create a pipeline in the UI:

1. Go to Discover ->Pipelines and click Add Pipeline.

2. Enter the pipeline details:

   • Name: A user-friendly, unique name
   • UID: A permanent, unique pipeline ID that cannot be changed later
   • Description: Optional
   • Owner: Optional
   • Team: Optional
   • Code Location: Optional. URL to the pipeline source code repository or definition file
   • Context: Optional key-value metadata
   • Tags: Optional. Used for classification and filtering

3. Click Save.

The pipeline now appears on the listing page.

You can also create a pipeline programmatically by using the Acceldata SDK or the Create Pipeline API.

To create a pipeline by using the SDK:

1. Install the Python SDK: acceldata-sdk
2. Generate API keys from Control Center -> Security -> API Keys
3. Initialize the SDK client with your ADOC URL and API keys
4. Define the pipeline object with a uid and name
5. Call create_pipeline()

If you use the SDK to report pipeline runs with a pipeline_uid, ADOC groups those runs under the matching pipeline. If the pipeline does not already exist, it is registered automatically when the run metadata is received.

See the Python SDK Guide and the SDK API Reference for details.

Run a Pipeline

Each execution of a pipeline is called a pipeline run. ADOC does not trigger pipeline runs directly. Runs are initiated by your orchestration tool, such as Airflow, Spark, or Fivetran, and the metadata is reported to ADOC through the SDK, a datasource connector, or OpenLineage.

When a run is reported, ADOC captures the following information:

• Run status: Whether the run succeeded, failed, is still running, or was cancelled
• Spans and events: Detailed timing and logs for each task or step within the run
• Asset nodes: Which data assets were read from or written to, along with their health metrics
• Policy compliance: Results of any data quality, schema drift, reconciliation, or other reliability checks executed during the run

To view runs for a specific pipeline, click the pipeline name on the listing page. This opens the Pipeline Run Details page, where you can inspect the lineage graph, timeline, events, and any triggered automations. You can also compare two runs by clicking the run timestamp dropdown and selecting a second run.

For SDK-based pipelines, you can create a run programmatically by calling create_pipeline_run() and passing the pipeline UID. See the SDK documentation for code examples.

Edit a Pipeline

You can update a pipeline’s metadata at any time without losing its run history.

To edit a pipeline:

1. On the Pipelines Listing page, find the pipeline you want to edit.
2. Click the ellipsis icon on the right side of the pipeline row.
3. Select the edit icon.
4. Update the fields you want to change, such as name, description, tags, owner, or team.
5. Click Save.

The pipeline UID cannot be changed after creation. If you need a different UID, you must delete the pipeline and create a new one.

Delete a Pipeline

To delete a pipeline from ADOC:

1. On the Pipelines Listing page, find the pipeline you want to remove.
2. Click the ellipsis icon on the right side of the pipeline row.
3. Select the delete icon.
4. Click Confirm in the confirmation dialog.

Deleting a pipeline permanently removes its registration and all associated run history from ADOC. This action cannot be undone. Deleting a pipeline from ADOC does not stop it from running in your orchestrator. It only removes observability tracking from ADOC.

Supported Pipeline Integrations

ADOC supports pipeline observability across multiple orchestration tools. These integrations fall into three categories:

• OpenLineage pipelines
• Datasource-based pipelines
• SDK-based pipelines

OpenLineage Pipelines

ADOC supports OpenLineage as a standard for pipeline event processing. OpenLineage is an open standard for collecting lineage and run metadata from data pipelines. ADOC processes these events to build pipeline views, lineage graphs, and run details.

OpenLineage is the preferred integration approach because it provides a seamless, low-effort, near real-time way to capture pipeline metadata in ADOC.

For setup and configuration details, see the integration documentation for:

• OpenLineage with Airflow
• OpenLineage with Spark
• OpenLineage with Trino

Datasource-Based Pipelines

The following integrations require you to configure a datasource in Register > Data Sources before pipeline data appears in ADOC.

• Fivetran: ADOC connects to Fivetran through its API to fetch pipeline and run data. Both account-level and individual Fivetran Pipeline Connector (FPC) connections are supported. When configuring the Fivetran datasource in ADOC, you select which FPC connections to monitor. The API key used in the datasource configuration must have CONNECTOR:READ access. The frequency at which ADOC captures new runs depends on the sync duration configured for each FPC connection in Fivetran.
• dbt Cloud: ADOC fetches dbt job details through the dbt Cloud API and builds pipeline and run information from that metadata. A datasource configuration is required. ADOC supports the following dbt model commands: run, build, test, seed, and snapshot.
• AutoSys: ADOC integrates with Autosys through a datasource configuration. Once connected, ADOC fetches job and pipeline metadata from Autosys for observability tracking.
• SnapLogic: ADOC integrates with SnapLogic through a datasource configuration. Pipeline run metadata is fetched from SnapLogic for monitoring in ADOC.
• Azure Data Factory: ADOC integrates with Azure Data Factory through a datasource configuration. Once connected, ADOC fetches pipeline and activity run metadata from ADF for observability tracking.

SDK-Based Pipelines

The following integrations use SDK instrumentation.

• Airflow: Acceldata provides a dedicated Airflow SDK, acceldata-airflow-sdk, with APIs, decorators, and operators for end-to-end tracking of Airflow DAGs. The SDK wraps the Airflow DAG to track pipeline execution, creates spans for each task, and supports job-level tracking with input and output asset information. In Airflow, the root span corresponds to the DAG run and each child span corresponds to a task instance.
• Spark: Acceldata supports Spark pipeline observability through SDK instrumentation. In Spark, the root span corresponds to the Spark application and each child span corresponds to a Spark job (action). The SDK captures execution timing, status, and asset information for each job within the application.

What’s Next

Once a pipeline has runs, click its name to open the run detail view. See Pipeline Run Details. To configure monitoring policies and automated data reliability checks, see Manage Pipelines.