Iceberg Catalog

Explanation (Core Concepts)

What is an Apache Iceberg Catalog?

An Apache Iceberg catalog registers Iceberg tables in xStore as a governed catalog. It is the most flexible catalog type: you choose a metadata backend (Hive Metastore, JDBC, or REST) and a storage layer (HDFS, S3, GCS, or ADLS) independently, each with its own authentication (dual-auth). This guide onboards an Iceberg catalog with a Hive Metastore backend on Kerberos and HDFS storage, and shows the JDBC and REST backend options too.

This guide is written for the Platform Administrator. It assumes a running xStore (<xstore-name>) with a metalake (<metalake-name>).

The steps and screenshots in this guide use placeholders, not real object names — substitute your own values wherever you see one:

Info

Related: this creates an Iceberg catalog inside a metalake via the standard Add Catalog wizard. For the built-in Iceberg REST catalog service (managing namespaces and tables that any Iceberg REST client can read), see xStore Iceberg REST.

Info

Security note: All screenshots mask connection details — the Hive Metastore host, HDFS namenode host, realm, and Kerberos principal are masked; uploaded files show only their filenames. Use your real values when you follow these steps.

Key Concepts

Info

New to xStore catalogs? See xStore Catalog Relational — Core Concepts for the shared model (metalake, catalog types, identity/run-as, the wizard, browsing).

Concepts specific to Iceberg:

• Catalog backend — where Iceberg table metadata is tracked: Hive (Hive Metastore over Thrift), JDBC (a relational DB), or REST (an external Iceberg REST server). The backend's authentication options change with the choice.

• Storage — where data files live: HDFS, S3, GCS, or ADLS. The storage authenticates independently of the backend (dual-auth). HDFS storage reuses the backend's Hadoop/Kerberos configuration (SIMPLE storage auth).

• Warehouse — the data root under which tables are written (e.g. hdfs://<namenode>:8020/warehouse/iceberg).

• Hive Service Principal — for a Kerberos Hive backend, the HMS service principal (e.g. hive/_HOST@<REALM>), read from hive-site.xml.

Tutorial (Getting Started)

This tutorial onboards an Iceberg catalog iceberg_catalog with a Kerberos Hive backend and HDFS storage into <metalake-name>.

Prerequisites

• A running xStore cluster and a metalake.

• A reachable Hive Metastore (Thrift URI) and HDFS warehouse path.

• For Kerberos: a principal, its keytab, and core-site.xml, hdfs-site.xml, hive-site.xml, krb5.conf.

• Host aliases for the HMS/namenode hostnames added to the xStore cluster's Network Settings.

• Administrative permissions on the xDP platform.

Minimum permissions

Enforced by the Hive Metastore + HDFS (and Ranger, if deployed) for the Kerberos principal the catalog authenticates as (and, because compute queries run as the per-user identity, the principal those users map to). References: Apache Ranger · HDFS Permissions Guide.

• Read-only: Hive Metastore metadata read; HDFS r-x / r-- on the warehouse + table paths.

• Read-write: the above plus HMS create/alter/drop table and HDFS rwx / rw- on the write paths.

Step 1 — Catalog Type & Provider

Open your metalake and click Add Catalog. Select Relational as the catalog type, then choose Apache Iceberg as the provider.

Step 2 — Basic Information & Identity

Set the Catalog Name (e.g. iceberg_catalog) and an optional comment, then accept the default Identity (System User → xstore-system-user).

Step 3 — Backend

Choose the Catalog Backend. This guide uses hive:

• URI: the Hive Metastore Thrift URI, e.g. thrift://<hms-host>:9083.

• Warehouse: the data root, e.g. hdfs://<namenode>:8020/warehouse/iceberg.

• Hive Service Principal (Kerberos): e.g. hive/_HOST@<REALM>.

• Authentication: KERBEROS → provide the Kerberos Principal and upload the Kerberos Keytab.

• Configuration Files: upload core-site.xml, hdfs-site.xml, hive-site.xml, and krb5.conf.

Other backends. The same step supports two more backends:

JDBC — track Iceberg metadata in a relational DB. Provide the JDBC URI, Warehouse, and JDBC user/password.

REST — point at an external Iceberg REST catalog server. Provide its URI and Warehouse.

Step 4 — Storage

Select the Storage Type — here HDFS. For HDFS, Storage Authentication is SIMPLE: it reuses the Hadoop/Kerberos configuration from the backend, so no separate storage credentials are needed. (S3 needs access keys, GCS a service-account JSON, ADLS an account key.)

Step 5 — Review & Create

Review the derived properties (catalog-backend, uri, warehouse, hive.metastore.kerberos.principal, kerberos.principal; host/principal masked here) and click Create Catalog.

The catalog is created as an lakehouse-iceberg / relational catalog. Open it to see its properties and browse its namespaces and tables.

How-to Guides

Use a JDBC or REST backend

1. In the Backend step, change Catalog Backend to jdbc or rest.

2. JDBC: provide the JDBC URI (e.g. jdbc:postgresql://<host>:5432/iceberg), warehouse, and JDBC user/password.

3. REST: provide the Iceberg REST server URI (e.g. http://<rest-host>:8181/api/catalog) and warehouse.

4. Pick the Storage that matches your warehouse URL and complete the wizard.

Use S3 / GCS / ADLS storage

1. In the Storage step, select S3, GCS, or ADLS.

2. Provide the storage credentials — S3 access key + secret, GCS service-account JSON, or ADLS account key — and a matching warehouse URL (s3a://…, gs://…, abfss://…).

Reference

Catalog backends

Storage layers

Configuration files (Kerberos Hive backend)

Best Practices

• Set host aliases on the cluster for the HMS and namenode hostnames your config files reference, or the catalog cannot connect.

• Match storage auth to the warehouse. HDFS storage reuses the backend Kerberos config (SIMPLE); cloud storage needs its own credentials.

• Scope HDFS/HMS access for the catalog's principal to the namespaces and paths you expose.

• Protect credentials — the keytab and config files are stored as secrets; never share them.

What's Next

• Browse namespaces and tables, or manage them via the Iceberg REST interface.

• Link an xCompute cluster so Spark/Trino can query the Iceberg tables.

• Onboard more catalogs (Snowflake, PostgreSQL, MySQL, Unity, fileset) into the same metalake.