OpenLineage with Airflow on AWS Managed Workflows for Apache Airflow (MWAA)

This guide explains how to configure OpenLineage integration with ADOC in an Airflow environment running on AWS Managed Workflows for Apache Airflow (MWAA).

Step 1: Add OpenLineage as a dependency

MWAA manages Python dependencies through the requirements.txt file. Add the following line alongside any existing dependencies:

You can specify a version, or leave it blank to install the latest available version.

Step 2: Configure OpenLineage transport

There are two options for configuring authentication between OpenLineage and ADOC. Configure one of the following.

Option 1: Use custom_headers

A token provider plugin is not required when using custom_headers.

Create a file named env_var_plugin.py with the following content. Replace the placeholders with your ADOC tenant values:

With this approach, only env_var_plugin.py is required. The token provider plugin and an Airflow Connection are not required.

Option 2: Use an Airflow Connection with a token provider

Use this option if you prefer to manage your ADOC access key and secret key through an Airflow Connection instead of specifying them directly in the transport configuration. This option requires the token provider plugin.

Create a file named access_key_secret_token_provider.py. Use the same token provider implementation described in OpenLineage with On-Premise Airflow.

Then create env_var_plugin.py with the following content:

Follow the steps below to create the Airflow Connection that provides your access key and secret key:

1. Log in to the Airflow UI.

2. Navigate to Admin → Connections.

3. Click Add (➕) to create a new connection.

4. Fill in the following details:

   • Connection ID: acceldata_connection

   • Connection Type: HTTP

   • Host: The host name of your ADOC tenant (use the same host value as defined in the transport configuration)

   • Login: Your ADOC access key

   • Password: Your ADOC secret key

5. Click Save to create the connection.

Once saved, Airflow securely manages your credentials, so you can omit them from the main transport configuration.

Step 3: Configure the plugin folder structure

The required plugin folder structure depends on the option you selected in Step 2.

Option 1: custom_headers

Place env_var_plugin.py in a single folder, preserving this directory structure before creating the zip archive:

Option 2: Airflow Connection + token provider

Place env_var_plugin.py and access_key_secret_token_provider.py into a single folder, preserving this directory structure before creating the zip archive:

Step 4: Package and upload the plugins

Option 1: custom_headers

From inside the folder, create a zip file containing env_var_plugin.py:

Option 2: Airflow Connection + token provider

From inside the folder, create a zip file containing env_var_plugin.py and the token provider:

Upload requirements.txt (from Step 1) and plugins.zip to the S3 bucket linked to your MWAA environment.

Step 5: Update the MWAA environment

1. Edit your MWAA environment settings.

2. Update the paths for plugins.zip and requirements.txt to point to the newly uploaded files.

3. Save the environment. MWAA restarts automatically to apply the changes.

Step 6: Verify the setup

After the environment becomes available, DAGs running in MWAA send OpenLineage events to the ADOC platform, enabling DAG monitoring and lineage tracking.

What's next

• To associate Airflow tasks that trigger dbt Cloud jobs with their pipeline runs, see Link Airflow tasks to dbt Cloud pipeline runs in ADOC.

• To confirm your datasets link correctly to Catalog assets, see Asset correlation for Redshift, Glue, Pub/Sub, and Iceberg.