Iceberg with Spark

Apache Iceberg is an open table format for huge analytic datasets. Iceberg adds tables to compute engines, including Spark, Trino, PrestoDB, Flink, Hive, and Impala, using a high-performance table format that works just like an SQL table.

User Experience

Iceberg avoids unpleasant surprises. Schema evolution works and won't inadvertently un-delete data. Users don't need to know about partitioning to get fast queries.

• Schema evolution: Supports adding, dropping, updating, or renaming operations without unintended consequences.

• Hidden partitioning: Prevents user errors that can lead to silently incorrect results or dramatically slow queries.

• Partition layout evolution: Updates the layout of a table as data volume or query patterns change.

• Time travel: Enables reproducible queries that use the same table snapshot or lets users easily examine changes.

• Version Rollback: Allows users to quickly correct problems by resetting tables to a good state.

Spark/Iceberg Compatibility Matrix

Choose the Iceberg Runtime for Your Spark Version

Use the Iceberg runtime JAR that matches your Spark version. The Iceberg runtime is compiled against Spark-specific analyzer and extension classes. Using a runtime built for a different Spark version can cause extension-loading errors, such as AbstractMethodError. For more information, see Troubleshooting.

For cloud storage with Spark 4.1.1, also add the required Iceberg bundle:

• AWS: iceberg-aws-bundle:1.11.0

• Azure: iceberg-azure-bundle:1.11.0

• Google Cloud: iceberg-gcp-bundle:1.11.0

Important considerations

• Spark 4.1.1 uses Scala 2.13. Use the _2.13 Iceberg runtime. Do not use a _2.12 runtime with Spark 4.1.1.

• Do not use the Spark 3.5 Iceberg runtime with Spark 4.1.1. This mismatch can cause an AbstractMethodError, including failures related to RewriteUpdateTableForRowLineage. For more information, see Troubleshooting.

• Use Iceberg 1.8.1 with Spark 3.3.3 when running on JDK 8. Iceberg 1.9 and later require JDK 11-compatible bytecode and can fail on JDK 8.

• Upgrade legacy Iceberg 1.4.x runtimes before using them for production workloads. Some older ODP mpack builds include Iceberg 1.4.x, which does not include fixes available in later releases.

Configure Spark Shell and Spark SQL with the Hive Metastore Catalog

Recommended: Use the Hive metastore catalog for ODP 3.3.6.5 and later.

Use SparkSessionCatalog with the Hive metastore to access Iceberg and standard Hive tables through the same spark_catalog namespace.

Configure Spark Shell with the following options:

Iceberg tables created with this configuration are registered in the Hive metastore and are available in Spark as:

Access Iceberg metadata tables

When you use SparkSessionCatalog, specify the fully qualified name when accessing Iceberg metadata tables.

For example:

Do not use the short table name:

Using the short form can result in a Table or view not found error.

Hive Metastore Compatibility

In releases earlier than ODP 3.3.6.5, using SparkSessionCatalog with type=hive can fail during the first Hive Metastore call with the following error:

This issue occurs because Hive 4 removed legacy Hive Metastore APIs, including get_table and get_table_objects_by_name, that are used by Spark's Hive 2.3 compatibility layer.

ODP 3.3.6.5 and later include a compatibility fix that restores the required APIs in the Hive 4 Metastore. No metastore version overrides or workaround configurations are required when you use SparkSessionCatalog with type=hive.

If this error occurs on ODP 3.3.6.5 or later, restart the Hive service from Ambari after installing the Mpack.

Configure Spark with HadoopCatalog

If the Hive Metastore (HMS) is unavailable or not required, use HadoopCatalog. This configuration is independent of HMS and is useful for smoke tests or environments that don't use HMS.

Tables created with this configuration are available under:

Import Statements

Include the necessary Apache Iceberg and Spark libraries to enable data operations.

Perform CRUD Operations on an Iceberg Table

After configuring the Iceberg catalog, use the following example to create an Iceberg table and perform insert, update, delete, and merge operations. The example applies to all supported Spark versions.

Create an Iceberg table

Create a partitioned Iceberg table in the default database.

Insert data

Insert sample records into the table.

Update data

Update the amount value for a specific record.

Delete data

Delete records for a specific region.

Merge data

Use MERGE INTO to update matching records or insert new records.

View snapshots

Query the Iceberg metadata table to view the available snapshots.

Run a time travel query

Use a snapshot ID to query an earlier version of the table.

For more details, see Apache Iceberg Getting Started Guide.

Configure Iceberg File Formats

Apache Iceberg supports Parquet, ORC, and Avro as storage file formats. Parquet is the default format. To use a different format, configure the write.format.default table property.\

Parquet

• Parquet is the default file format for Iceberg tables and does not require additional configuration.

• The CRUD examples in the previous section use Parquet by default.

ORC

To store an Iceberg table in ORC format, set the write.format.default table property to orc when you create the table.

Insert data into the ORC-backed Iceberg table:

This configuration is validated with the recommended Iceberg runtimes for Spark 3.3.3, 3.5.1, 3.5.5, and 4.1.1.

Avro

To use Avro as the storage format, set the write.format.default table property to avro:

Avro is supported by the Iceberg runtime but was not validated with ODP 3.3.6.5.

Troubleshooting

Use the following information to troubleshoot common Apache Iceberg issues with Spark.

Resolve AbstractMethodError on Spark 4.1.1

On Spark 4.1.1, an AbstractMethodError can occur if the Iceberg runtime for Spark 3.5 is present on the classpath together with the Iceberg runtime for Spark 4.1.

The following error can occur while Spark loads the Iceberg extensions:

Cause

The iceberg-spark-runtime-3.5_2.13 runtime is compiled against Spark 3.5 and is not compatible with the Spark 4.1 row-level command implementation.

This error occurs when the Spark 3.5 Iceberg runtime is loaded on the Spark 4.1.1 classpath. Use only the Iceberg runtime that corresponds to Spark 4.1.1.

Resolution

Remove the Spark 3.5 Iceberg runtime from the Spark 4 classpath.

If the JAR is present in /usr/odp/current/spark4-client/jars/, move it to a different location and add the Spark 4.1 runtime:

Rerun the Spark job after replacing the JAR. A Spark restart is not required.

For a permanent resolution, upgrade to an Mpack version that includes only the Spark 4.1 Iceberg runtime in the Spark 4 assembly.

Resolve "Table or view not found" for Metadata Tables

When you use SparkSessionCatalog, accessing an Iceberg metadata table by its short name can result in a Table or view not found error.

For example, do not use:

Instead, specify the fully qualified catalog, database, and table name:

When you use SparkSessionCatalog, specify metadata tables as:

For example:

SparkCatalog configured with type=hadoop accepts the short form, while SparkSessionCatalog requires the fully qualified name.

Resolve NoClassDefFoundError During Extension Loading

A NoClassDefFoundError related to row-level operation classes can occur when the Iceberg runtime does not match the Spark version.

Ensure that the Spark and Iceberg runtime versions correspond:

• Spark 3.3 uses the Iceberg runtime for Spark 3.3.

• Spark 3.5 uses the Iceberg runtime for Spark 3.5.

• Spark 4.1 uses the Iceberg runtime for Spark 4.1.

For the runtime required for each supported Spark version, see Choose the Iceberg Runtime for Your Spark Version.

Resolve Iceberg Bytecode Version Errors on Spark 3.3

When building ODP Spark 3.3 from source, EnforceBytecodeVersion can fail if the configured Iceberg runtime requires a newer Java bytecode version.

To resolve the issue, use one of the following approaches:

• Use iceberg-spark-runtime-3.3_2.12:1.8.1, which is the last Spark 3.3 runtime in this source guidance that targets JDK 8.

• If the build is intended to use JDK 11, set <java.version> to 11 in the POM and add the following exclusion under <enforceBytecodeVersion>:

Use the same Iceberg exclusion pattern as other Iceberg exclusions defined in the parent POM.

Resolve "Invalid method name: 'get_table'"

In releases earlier than ODP 3.3.6.5, creating an Iceberg table using SparkSessionCatalog with type=hive can fail with the following error:

ODP 3.3.6.5 and later include the Hive Metastore compatibility fix required by Spark.

If this error occurs on ODP 3.3.6.5 or later, restart HiveServer2 from Ambari and retry the operation.

For more information, see Hive Metastore Compatibility.