OpenLineage Asset Correlation

Pipeline lineage links OpenLineage datasets to Catalog assets when the job emits OpenLineage events (START plus COMPLETE or FAIL) with inputs and outputs, each dataset follows the OpenLineage naming spec, and the same asset is already onboarded and crawled in ADOC.

If the format is unsupported or the Catalog asset is missing, the dataset still appears in the pipeline graph, but it is not linked to the Catalog.

Summary

Onboard the matching data source in ADOC first. The event must identify the same catalog, cluster, project, or bucket.

Source	Namespace	Name	Also required
Iceberg (REST)	often `s3://…`	optional	`catalog` (`framework=iceberg`, `type=rest`) + TABLE symlink
Iceberg (Glue)	often `s3://…`	optional	`catalog` (`framework=iceberg`, `type=glue`) + Glue TABLE symlink
AWS Glue table	`arn:aws:glue:region:account`	`schema.table`	TABLE symlink if the primary is an S3 path
Redshift	`redshift://cluster.region:port`	`schema.table`	—
Snowflake	`snowflake://account`	`DATABASE.SCHEMA.TABLE`	—
BigQuery	`bigquery://`	`project.dataset.table`	—
Trino	`trino://host:port`	`catalog.schema.table`	—
Databricks Unity Catalog	often `s3://…`	optional	`catalog.type=unity` and/or `unity-catalog` TABLE symlink
Pub/Sub	`pubsub`	`topic:project:topic-id`	Topics only
S3 / GCS / ADLS	`s3://`, `gs://`, `abfss://`	object path	Onboarded path expression (glob) must cover that path

When a TABLE symlink is present, ADOC uses that identity (not the S3 path).

Not Linked Yet PostgreSQL, MySQL, MariaDB, Oracle, SAP HANA, SQL Server, Kafka, Hive (non-Glue), Presto, MongoDB, Cassandra, Teradata, DB2, ClickHouse, Athena, Databricks Hive Metastore, Iceberg Hive/Nessie/JDBC/Hadoop catalogs, Pub/Sub subscriptions, JDBC URLs as namespace, and file:// / wasbs:// / dbfs:// paths.

Details

Iceberg

Iceberg is identified by the catalog facet, not by the S3 path. Every Iceberg input/output must include:

facets.catalog with framework: iceberg and type of rest or glue
facets.symlinks with a TABLE identifier for that catalog

Without those facets, an s3:// namespace is treated as a file, not an Iceberg table. Do not use iceberg:// as the dataset namespace. Only REST and Glue Iceberg catalogs are supported.

REST catalog (`type: rest`)

Onboard the Iceberg REST catalog in ADOC. metadataUri and the TABLE symlink namespace must be that same catalog URL. Table name: demo.orders (or table/demo/orders).

{ "namespace": "s3://my-bucket", "name": "any-path", "facets": { "catalog": { "framework": "iceberg", "type": "rest", "name": "spark_catalog", "metadataUri": "https://iceberg-catalog.example.com/spark_catalog/" }, "symlinks": { "identifiers": [{ "namespace": "https://iceberg-catalog.example.com/spark_catalog/", "name": "demo.orders", "type": "TABLE" }] } } }

Glue catalog (`type: glue`)

Onboard Iceberg on AWS Glue in ADOC. The TABLE symlink namespace is the Glue catalog ARN; its region must match the onboarded connection. Table name: sales.orders or table/sales/orders.

{ "namespace": "s3://my-bucket", "name": "optional", "facets": { "catalog": { "framework": "iceberg", "type": "glue", "name": "glue_catalog" }, "symlinks": { "identifiers": [{ "namespace": "arn:aws:glue:us-east-1:123456789012", "name": "table/sales/orders", "type": "TABLE" }] } } }

AWS Glue tables

Hive / Spark tables without Iceberg facets. Do not put the table path in the Glue ARN, and do not omit the arn:aws:glue: prefix.

{ "namespace": "arn:aws:glue:us-east-1:123456789012", "name": "sales.orders" }

If the job only reports an S3 path, add a Glue TABLE symlink:

{ "namespace": "s3://my-bucket", "name": "data/", "facets": { "symlinks": { "identifiers": [{ "namespace": "arn:aws:glue:us-east-1:123456789012", "name": "table/sales/orders", "type": "TABLE" }] } } }

Amazon Redshift

Use redshift://cluster.region:port — not a JDBC URL. Cluster, region, and port must match the onboarded Redshift connection.

{ "namespace": "redshift://my-cluster.us-east-1:5439", "name": "public.orders" }

Snowflake

Use snowflake://… (account locator or Snowflake host), not a JDBC URL. name must be DATABASE.SCHEMA.TABLE. The account must match the onboarded Snowflake connection.

{ "namespace": "snowflake://myorg-myaccount", "name": "PRODUCTION_DB.ANALYTICS.USER_EVENTS" }

BigQuery

name must be project.dataset.table. The project must match the onboarded BigQuery connection.

{ "namespace": "bigquery://", "name": "my-project.analytics_dataset.user_events" }

Trino

Host and port must match the Trino instance onboarded in ADOC.

{ "namespace": "trino://coordinator.example.com:8080", "name": "catalog.schema.orders" }

Databricks Unity Catalog

The table must already be crawled from that Databricks workspace. Databricks Hive Metastore (non–Unity Catalog) is not supported.

{ "namespace": "s3://databricks-workspace-bucket", "name": "optional", "facets": { "catalog": { "framework": "delta", "type": "unity", "name": "analytics" }, "symlinks": { "identifiers": [{ "namespace": "unity-catalog", "name": "analytics.default.employee_processed_ext", "type": "TABLE" }] } } }

Google Cloud Pub/Sub

Topics only. Subscriptions do not link. name must start with topic:. The GCP project and topic must match the onboarded Pub/Sub source.

{ "namespace": "pubsub", "name": "topic:my-gcp-project:orders-events" }

Object storage (S3, GCS, Azure Data Lake)

Supported schemes: s3://, gs:// / gcs://, and abfss://. If the same dataset also has a Glue, Iceberg, or Unity Catalog TABLE symlink, ADOC links the table, not the file path.

{ "namespace": "s3://my-bucket", "name": "data/events.parquet" }

ADOC joins namespace and name into one path (for example s3://my-bucket/data/events.parquet) and links it only when both are true:

The bucket (S3/GCS) or storage account (ADLS) matches the onboarded file source.
That path matches the file asset path expression you configured when onboarding (glob, not Java regex).

Use the same URI scheme in the event as in the onboarded path expression (s3:// vs s3n://, gs:// vs gcs:// do not match each other).

Path expression glob

Matching is per path segment (/-separated). Wildcards: * (any characters in one segment), ? (one character). A trailing * matches any remaining depth. ** is not a recursive glob — it behaves like a single *.

Onboarded path expression	Event path	Links?
`s3://my-bucket/data/events.parquet`	`s3://my-bucket/data/events.parquet`	Yes — exact
`s3://my-bucket/data/*.parquet`	`s3://my-bucket/data/events.parquet`	Yes — `*` in that segment
`s3://my-bucket/data/incoming/*`	`s3://my-bucket/data/incoming`	Yes — trailing `*`
`s3://my-bucket/data/raw/*`	`s3://my-bucket/data/raw/sales/date=2026-04-01/file.parquet`	Yes — trailing `*` is greedy
`s3://my-bucket/data/raw/sales`	`s3://my-bucket/data/raw/sales/date=2026-04-01`	Yes — event may be one segment deeper
`s3://my-bucket/data/raw/sales/ingestion_date=*`	`s3://my-bucket/data/raw/sales/ingestion_date=2026-04-01`	Yes — partition wildcard
`s3://my-bucket/data/raw/sales/ingestion_date=/.parquet`	`s3://my-bucket/data/raw/sales/ingestion_date=2026-04-01`	Yes — extra file-level glob is optional
`s3://my-bucket/data/*/incoming`	`s3://my-bucket/data/landing/incoming`	Yes — `*` in the middle
`s3://my-bucket/data/cleansed/date=????-??-??`	`s3://my-bucket/data/cleansed/date=2026-04-01`	Yes — `?` is one character
`s3://my-bucket/data/raw/sales`	`s3://my-bucket/data/raw/sales/date=2026-04-01/file.parquet`	No — two extra segments without a trailing `*`
`s3://my-bucket/data/raw/**`	`s3://my-bucket/data/raw/sales/date=2026-04-01/file.parquet`	No — `**` is not recursive
`s3://my-bucket/data/raw/sales/ingestion_date=2025-01-01`	`s3://my-bucket/data/raw/sales/ingestion_date=2026-04-01`	No — different partition
`s3://my-bucket/data/raw/other_table/ingestion_date=*`	`s3://my-bucket/data/raw/sales/ingestion_date=2026-04-01`	No — different prefix
`s3://other-bucket/data/events.parquet`	`s3://my-bucket/data/events.parquet`	No — bucket / account must match
`s3n://my-bucket/data/*.parquet`	`s3://my-bucket/data/events.parquet`	No — scheme must match
`file://` / `wasbs://` / `dbfs://` event	any onboarded object store	No — those schemes are not supported

GCS uses the same rules with gs:// (or gcs:// on both sides). ADLS uses abfss://container@account.dfs.core.windows.net/…; the storage account must match the onboarded source.

If the asset does not link

Confirm the source is in the summary table (not the unsupported list).
Confirm the asset is onboarded and crawled in ADOC, on the same connection (catalog URL, cluster, project, region, or bucket).
Compare the event namespace, name, and facets with the example for that source.
For Iceberg and Unity Catalog, confirm catalog + TABLE symlinks are present. An S3-only dataset is treated as a file, not a table.

What's next

For Spark-specific OpenLineage setup, see Spark OpenLineage Integration.
For dbt-specific setup, see dbt Core OpenLineage Integration.

Last updated on

Was this page helpful?