dbt Core OpenLineage Integrations

dbt lets you define, test, and document SQL-based transformations, typically against cloud data warehouses, to create reusable, analytics-ready datasets. This page explains dbt Core's key concepts and how to configure OpenLineage so ADOC can track dbt pipeline runs and lineage.

dbt Core concepts

• Models: SQL files that transform raw data into analytics-ready datasets. Models are the core building block in dbt.

• Tests: Validate data transformations.

• Seeds: CSV files loaded into your data warehouse, used as static reference data.

• Macros: Reusable Jinja templates that promote code reuse across models, support control flow, and enable complex transformations.

How ADOC integrates with dbt

ADOC uses OpenLineage to receive dbt events and build pipelines for lineage tracking and observability. In this pattern, dbt with OpenLineage configured in your environment sends events to ADOC Pipelines, which stores and processes them.

Environment setup

Step 1: Create a Python virtual environment

Step 2: Install required libraries

Step 3: Clone a sample dbt project (optional)

To follow this guide with a working example, you can clone a public sample project:

Configuration

Step 1: Configure the Snowflake connection

Create a profiles.yml file for the dbt-Snowflake connection, and update the details for your environment:

Step 2: Configure OpenLineage

Add an openlineage.yml configuration file and include the API key from ADOC:

Add an environment variable pointing to the configuration file path:

Set up Snowflake

If you're using the sample project, create the initial tables in Snowflake by running the following script in the desired database. Ensure the database name matches the one in profiles.yml:

Run dbt

Step 1: Load demo data

Step 2: Run models with OpenLineage

Use dbt-ol instead of dbt to generate OpenLineage events when running models:

Step 3: Run tests

Once ADOC Pipelines receives the dbt events, it automatically detects the dbt job and creates the lineage.

dbt Core with Airflow

Install OpenLineage on the Airflow worker

Ensure the openlineage-dbt library is installed on the Airflow worker:

Sample Airflow DAG with a dbt operator

Note: When dbt Core runs through Airflow this way, OpenLineage does not provide complete end-to-end lineage. If your Airflow DAG triggers dbt Cloud jobs (rather than dbt Core), see Link Airflow Tasks to dbt Cloud Pipeline Runs in ADOC instead.

What's next

• To confirm your dbt datasets link correctly to Catalog assets, see Asset Correlation for Redshift, Glue, Pub/Sub, and Iceberg.

• For an overview of how ADOC interprets OpenLineage events generally, see OpenLineage Integration.