xDP 6.0.0

Highlights

• New catalog providers in xStore — MySQL, Oracle and Unity Catalog.
• Column masking and row-level filter policies in Ranger governance.
• Governed YuniKorn queues — capacity management with submit and admin entitlements.
• Kafka as a managed app — operator-driven install with broker version and topology choices.
• NiFi as a managed app — NiFi 1.x is generally available, NiFi 2.x is a preview.
• Multi-version app install — one app can publish several version lines.
• Dynamic job arguments — {{{NAME}}} placeholders resolved at run time.
• Trino configuration and tuning in the UI, with versioned history.
• Service Health dashboard — one view of the control plane and every data plane.
• Data plane resource pages — secrets, config maps, backups, Helm releases and pod logs.
• Product analytics — optional, off-by-default usage telemetry.
• xStore high availability — runs highly available by default and scales with load.
• Upgraded catalog engine — xStore now runs on Apache Gravitino 1.3.0.
• Enhanced security — fixes for known vulnerabilities across xDP and its infrastructure.

What's New

1. New catalog providers in xStore

• MySQL, Oracle and Unity Catalog join the catalog providers xStore can register, so tables and schemas that already exist in those systems become browsable and governable from xDP without a migration.
• All three appear in the Data catalog explorer and can be governed with policies, like any other catalog.

2. Governance — column masking and row-level filters

• Column masking policies — scope to a table and its columns, and pick a mask type per user or group. Mask types are validated against the column's data type before the policy is saved.
• Row-level filter policies — scope to a table and attach a filter expression per user or group.
• Custom expression-defined masks are no longer offered; use the built-in mask types.

3. Resource management with YuniKorn

• Queue management under Infrastructure → Queues: browse the queue hierarchy and edit guaranteed and maximum capacity. Edits are staged as a draft and applied as a reviewed set.
• Queue access control: submit and admin entitlements per queue for users and groups, stored as Ranger policies and enforced at job submission. admin implies submit.

4. Kafka integration

• Operator-driven Kafka install with broker version selection — 3.7.2 (KRaft, recommended) or 2.8.2 (ZooKeeper, legacy).
• Topology configuration: metadata mode, broker count, quorum size, and an HA switch that raises the minimum to three brokers.
• Optional Kafka UI and MirrorMaker 2, toggled at install.

5. NiFi integration

• NiFi 1.x is generally available on the v1 version line. It installs as an operator-driven managed app with a NiFi Registry and a NiFi proxy, and registers with Ranger so NiFi and NiFi Registry resources are governable.
• NiFi jobs support dynamic arguments (see §7), substituting into parameter values.
• NiFi 2.x is a preview on its own v2 version line and is not recommended for production. It runs without ZooKeeper and versions flows in a separate NiFi Registry 2.x.

6. Multi-version app install

• An app can publish several version lines, each with its own chart and upgrade path — for example Trino v476, NiFi v1 and v2.
• The install wizard gained an App Version selector alongside release version and chart version.
• A release version can carry per-chart versions, instead of one version fanned across every chart.

7. Dynamic arguments for jobs and workflows

• Declare variables on a job and reference them as {{{NAME}}} in its arguments. Supported on Spark, Notebook and Application (Java and Python) and NiFi.
• A value supplied at execute wins; otherwise the declared default applies. The same path serves Run now, scheduled runs and workflow tasks, and a workflow task can pass values through Airflow XCom.
• Each run records the resolved value and its origin per variable.

8. Trino resource configuration and tuning in the UI

• Edit config.properties, jvm.config and log.properties for the coordinator and worker separately, at install and afterwards from the app's configuration sheet.
• Versioned history — diff two versions, annotate, restore or delete a version.

9. Service Health dashboard

• A single page at Utilities → Service Health covering control-plane services and dependencies, and every registered data plane with its up-check, synthetic checks and per-app results.
• A data plane that cannot be reached is reported as UNREACHABLE with the error, rather than folded into a healthy count.

10. Data plane resources in the xDP UI

• New pages under Utilities, scoped to a data plane and namespace: Secrets, ConfigMaps, Backups, Helm Releases and Pod Logs.
• The Secrets page lists each secret's name, type, labels, age and its data key names — key names only, never the secret values. Open a secret to see its full list of data keys and their types, and search matches on both the secret name and its key names.

11. Product analytics

• xDP can emit anonymous product-usage telemetry to PostHog. It is off by default and initialises only when the posthog feature flag is on and a project key is configured. Session recording is disabled.

12. xStore high availability

• xStore now runs highly available by default. More than one instance serves the service, so a single instance failing or restarting no longer interrupts catalog access.
• Request handling and background processing run separately, so heavy background work no longer slows down catalog queries.
• xStore scales out on its own as load rises and scales back in when it falls.
• During upgrades and node maintenance, enough instances are kept running to stay available throughout.

13. Upgraded catalog engine

• The catalog engine behind xStore is upgraded to Apache Gravitino 1.3.0, bringing its latest catalog, schema and metadata improvements to xStore.

14. Security

• Enhanced security across xDP and its infrastructure components, including fixes for known vulnerabilities.