Asset Correlation for Redshift, Glue, Pub/Sub, and Iceberg

Pipeline lineage links to Catalog assets when your job emits OpenLineage events (START plus COMPLETE or FAIL) with input and output datasets, and the target asset is already onboarded in ADOC. Each dataset needs a namespace, a name, and sometimes facets.

Use this guide to confirm what your emitter should produce, and what ADOC checks when matching an event dataset to a Catalog asset. If lineage does not link as expected, compare your event payload against the tables in the relevant section below.

Naming reference: OpenLineage dataset naming spec

Airflow inlets and outlets

If you declare datasets on Airflow operators using inlets or outlets, use the OpenLineage Dataset type so events carry a spec-compliant namespace, name, and facets:

from openlineage.client.event_v2 import Dataset # not: from airflow.datasets import Dataset

Do not use Airflow's scheduling Dataset type (airflow.datasets.Dataset) — it produces a different URI shape and will not correlate to Catalog assets. Airflow's scheduling Dataset is for DAG scheduling dependencies only. Build each OpenLineage Dataset with the namespace and name (and facets when needed) from the source-specific sections below.

When correlation succeeds

ADOC links a pipeline dataset node to a Catalog asset when all of the following are true:

The dataset format matches one of the supported source patterns in this guide.
A Catalog assembly exists for that source type and has been crawled.
The connection properties on the assembly match what ADOC derives from the event.
The Catalog asset UID matches what ADOC derives from the event name (or symlink name).

If any of these checks fail, the pipeline node is still created, but without a linked Catalog asset.

How ADOC picks the source type

For datasets with symlinks that have a TABLE identifier, ADOC correlates using the TABLE symlink's namespace and name, not the top-level values. This is required for:

Iceberg tables whose primary namespace is an S3 path.
Glue tables emitted with S3-only primaries.

A valid TABLE symlink has type: "TABLE" and a namespace ADOC can resolve — a Glue ARN, a REST catalog URL, or another supported scheme.

For most sources, ADOC checks two things from each dataset:

Check	Meaning
Connection match	The assembly's connection property must equal (or regex-match) a value derived from the event
Asset UID match	The crawled asset UID must end with the table or topic identifier derived from the event

Pub/Sub uses a dedicated, topic-based match (see the Pub/Sub section below).

Apache Iceberg

Emit catalog and symlinks facets on each table input and output. The top-level namespace and name can be an S3 path — the facets define the table.

Do not use iceberg:// as a namespace for facet-based events. Iceberg is identified by facets, not a URI scheme.

Iceberg REST catalog

Emit:

{ "namespace": "s3://my-bucket", "name": "any-path-or-placeholder", "facets": { "catalog": { "framework": "iceberg", "type": "rest", "metadataUri": "https://iceberg-catalog.example.com/spark_catalog/" }, "symlinks": { "identifiers": [{ "namespace": "https://iceberg-catalog.example.com/spark_catalog/", "name": "demo.orders", "type": "TABLE" }] } } }

Validate against the catalog:

Event field	Must match catalog
facets.catalog.framework	iceberg
facets.catalog.type	rest
facets.symlinks TABLE namespace	https:// or http:// REST catalog URL (same catalog as metadataUri)
facets.symlinks TABLE name	Table name as demo.orders (or table/demo/orders, normalized to demo.orders)
facets.catalog.metadataUri	Assembly connection `iceberg.httpuri` = `https://iceberg-catalog.example.com/spark_catalog/`

Example asset UID: my-iceberg-assembly.demo.orders

Iceberg on AWS Glue

Emit:

{ "namespace": "s3://my-bucket", "name": "optional", "facets": { "catalog": { "framework": "iceberg", "type": "glue" }, "symlinks": { "identifiers": [{ "namespace": "arn:aws:glue:us-east-1:123456789012", "name": "table/sales/orders", "type": "TABLE" }] } } }

Validate against the catalog:

Event field	Must match catalog
facets.catalog.framework	iceberg
facets.catalog.type	glue
TABLE symlink namespace	`arn:aws:glue:us-east-1:123456789012`
TABLE symlink name	sales.orders or table/sales/orders
Region in the Glue ARN	Assembly connection `aws.region` = `us-east-1`

Example asset UID: my-iceberg-glue-assembly.sales.orders

Iceberg vs. plain Glue

What you see in the event	ADOC treats it as
catalog.framework=iceberg plus a valid TABLE symlink	Iceberg
Glue ARN namespace, no Iceberg catalog facet	AWS Glue
S3 primary plus Glue TABLE symlink, no Iceberg facet	AWS Glue
S3 primary, no symlinks, no Iceberg facet	File path — out of scope, no Catalog link

AWS Glue tables (Hive or Spark, not Iceberg)

Use this pattern when the job reads or writes a Glue Data Catalog table without Iceberg facets.

Direct Glue ARN (preferred)

Emit:

{ "namespace": "arn:aws:glue:us-east-1:123456789012", "name": "sales.orders" }

name can also be table/sales/orders.

Validate against the catalog:

Event field	Required pattern	Must match catalog
namespace	`arn:aws:glue:us-east-1:123456789012`	—
name	`sales.orders` or `table/sales/orders`	Asset UID `my-glue-assembly.sales.orders`
Region in ARN	`us-east-1`	Connection `aws.region` = `us-east-1` (exact)
Catalog source type	—	AWS Glue

Invalid formats (no correlation):

Fused ARN — table path embedded in the namespace, for example arn:aws:glue:us-east-1:123456789012:table/sales/orders as the namespace.
Non-ARN prefix — for example aws:glue:us-east-1:123456789012 instead of arn:aws:glue:us-east-1:123456789012.

S3 primary with a Glue symlink

Use this pattern when the emitter reports only S3 paths. Add a symlinks facet pointing at the Glue table:

{ "namespace": "s3://my-bucket", "name": "data/", "facets": { "symlinks": { "identifiers": [{ "namespace": "arn:aws:glue:us-east-1:123456789012", "name": "table/sales/orders", "type": "TABLE" }] } } }

ADOC uses the TABLE symlink for correlation, following the same Glue ARN and name rules as above. Do not include an Iceberg catalog facet unless the table is actually Iceberg.

Amazon Redshift

Use the OpenLineage warehouse naming form — typically produced automatically by Airflow or Spark Redshift integrations.

Emit:

{ "namespace": "redshift://my-cluster.us-east-1:5439", "name": "public.orders" }

Validate against the catalog:

Event field	Required pattern	Must match catalog
namespace	`redshift://my-cluster.us-east-1:5439`	Connection `redshift.url` (regex)
name	`public.orders`	Asset UID `my-redshift-assembly.prod.public.orders`
Cluster	`my-cluster` (first segment before the first `.`)	Part of the JDBC hostname
Region	`us-east-1`	Part of the JDBC hostname
Port	`5439`	JDBC port in the connection URL

ADOC builds the following connection regex from the namespace:

redshift(:iam)?://my-cluster\.[^.]+\.us-east-1\.redshift(-serverless)?\.amazonaws\.com(:5439)?/.*

Do not use jdbc:redshift://… as the dataset namespace — that form is not recognized as Redshift and will not correlate.

Google Cloud Pub/Sub

ADOC links topics only. The Catalog asset is the Pub/Sub topic — subscription names do not link, even when the job reads through a subscription.

Emit the topic your job consumes or produces:

{ "namespace": "pubsub", "name": "topic:my-gcp-project:orders-events" }

Validate against the catalog:

Event field	Required pattern	Must match catalog
namespace	`pubsub` (exact)	—
name	`topic:my-gcp-project:orders-events`	Topic ID = `orders-events`
Project	`my-gcp-project` (middle segment)	Connection `projectId` = `my-gcp-project` (exact)
Topic ID	`orders-events` (third segment)	Listed in connection `pubsub.topics.0` = `orders-events`, and assembly `pubsub.assets.0.topic.name` = `orders-events`

Example asset UID: my-pubsub-assembly.orders-events

Does not correlate:

Event name	Why
`subscription:my-gcp-project:orders-sub`	Subscriptions are not Catalog assets
`my-gcp-project:orders-events` (missing `topic:` prefix)	Malformed — falls back to the OpenLineage UID only
`topic:wrong-project:orders-events`	projectId does not match the connection

Validation checklist

Before testing, confirm:

A Catalog assembly is created and crawled for the target source.
Connection properties are populated as described in each "Validate against catalog" table above.
OpenLineage events follow the naming spec.
Iceberg Spark jobs include catalog and symlinks facets on table inputs and outputs.

If lineage does not link

Symptom	Check in the event	Check in the catalog
Node exists but has no assetId	Dataset format matches a section above	Asset crawled and assembly exists
Glue table shows as S3 only	Add symlinks TABLE with the Glue ARN, or Iceberg facets	Source type is AWS Glue or Iceberg, as appropriate
Iceberg-on-Glue treated as plain Glue	`catalog.framework` must be `iceberg`	Source type is Iceberg (AWS_GLUE), not plain Glue
Pub/Sub not linking	`name` must be `topic:…`, not `subscription:…`	projectId, pubsub.topics.0, and pubsub.assets.0.topic.name all match
Redshift not linking	namespace must be `redshift://…`, not `jdbc:redshift://…`	redshift.url regex matches the cluster, region, and port
Iceberg REST not linking	metadataUri in the event	iceberg.httpuri on the connection equals the same URL
Glue / Iceberg-on-Glue not linking	Region in the Glue ARN	aws.region on the connection matches

What's next

For Spark-specific OpenLineage setup, see Spark OpenLineage Integration.
For dbt-specific setup, see dbt Core OpenLineage Integrations.

Last updated on

Was this page helpful?