Pipelines Guide

This guide describes how to record pipeline executions in ADOC with acceldata-sdk-python.

Introduction

ADOC treats data pipelines as first-class observability objects. Each definition captures how work is structured (assets and jobs), and each run captures what happened during a given execution. acceldata-sdk-python creates those definitions, opens runs, attaches hierarchical spans, emits metrics and custom events, and closes runs with a clear outcome, so the Catalog and Pipelines views in ADOC stay aligned with your orchestration code.

Core Concepts

Pipeline APIs focus on orchestration observability: definitions, runs, spans, jobs, and events. Catalog APIs focus on what is stored and how it is profiled or crawled. Most production setups use both: you register and profile assets through catalog calls, while pipeline calls describe how those assets are produced or consumed in each batch or stream.

The rest of this guide is organized by topic: definition, runs, spans, jobs, events, completion, and retrieval. A full execution typically defines or loads a pipeline, opens a run, records spans and jobs, emits events as work proceeds, and then updates the run to a terminal state.

Creating a Pipeline

Use CreatePipelineInputRequest to define and create a pipeline in ADOC. This request includes required identifiers along with several optional fields for metadata and organization.

Key Fields

Tags (Optional)

Use the tags field to attach metadata for discovery and display in the ADOC UI.

If you don't provide tags, ADOC creates the pipeline without any tags.

Replacing Tags

Pipeline Runs

PipelineRunResource exposes helpers such as create_root_span(), get_root_span(), and update_pipeline_run().

Spans: Root Span and Hierarchy

Creating a root span automatically initializes it with the current timestamp. You can attach additional metadata to the span by sending events.

Child Spans (SpanResource.create_child_span)

Span Helpers and Run Completion

SpanResource exposes is_root(), has_children(), end(), abort(), and failed() to mirror orchestration state. Use GenericEvent for domain metrics and LogEvent for log lines. On failure, prefer failed() or abort() over end(). Close the root span and the run when the work finishes.

Jobs

pipeline.create_job and pipeline_run.create_job perform the same operation; only the run binding differs. On PipelineResource, CreateJobInput.pipeline_run_id is required. On PipelineRunResource, it is optional and defaults to that run's ID (when you set it, it must match).

CreateJobInput Fields

Example Usage

Use bounded_by_span=True to create a job and its span together, then fetch the span by UID.

Alternative with bounded_by_span=False — create the span explicitly and associate the job UID:

Set asset_uid on JobInputOutputRef(asset_uid=...) by copying the value from the ADOC UI.

Child Spans Under a Job

Child spans under a job span provide phase-level tracking (extract, transform, publish, and so on) while keeping the job as the parent context.

Span Events

LogEvent carries log-style text. GenericEvent carries a stable event_uid plus arbitrary context_data for domain metrics or checkpoints.

Ending Spans

Call end() when the span's work finishes normally. Use failed() or abort() to record an error or cancellation instead of a clean completion.

Completing the Pipeline Run

update_pipeline_run() returns the same PipelineRunResource with an updated payload. It accepts both result and status:

• status is the current lifecycle state of the run (for example, STARTED, COMPLETED, FAILED, ABORTED).

• result is the final business outcome (for example, SUCCESS or FAILURE) and applies when the run is terminal.

• When you omit both, the SDK defaults to result=SUCCESS and status=COMPLETED.

While work is still in flight, keep status=STARTED and typically result=RUNNING. When the run succeeds, set status=COMPLETED and result=SUCCESS. On failure, use status=FAILED with result=FAILURE.

Retrieve and Manage Existing Pipelines and Runs

Resolve a Pipeline (client.get_pipeline)

pipeline_identity is the string the server accepts to locate a single pipeline — typically the pipeline UID from creation, or the pipeline ID as a string when only the numeric key is available. The returned PipelineResource exposes create_pipeline_run, get_latest_pipeline_run, get_runs, and create_job (with an explicit pipeline_run_id on the input).

Get Latest Pipeline Run

Get Pipeline Details for a Run

Get All Spans for a Run

Get Runs for a Pipeline

Get All Pipelines

get_pipelines() returns a listing object. Iterate pipelines for each row, including pipeline_summary with uid and name.

Delete a Pipeline

Common Mistakes

Jobs and span events require an active run and open spans. Creating jobs before a run exists, emitting events after a span has ended, or reusing job or span UIDs within the same run all produce confusing failures. Leaving the pipeline run without a terminal update_pipeline_run call also leaves ADOC with an ambiguous outcome.

What's Next

After you complete this section, explore:

• End-to-End Pipeline Lifecycle Example Using acceldata-sdk-python – See a complete script that creates a pipeline, runs it, and closes it out.

• Pipeline continuation ID (multi-script runs) – Learn how multiple scripts or services can update the same pipeline run.

• Historical pipeline instrumentation with explicit times – Learn how to backfill past pipeline executions with explicit timestamps.

• Datasources and Assets Guide – Learn how job inputs and outputs reference catalog assets.