Access Hive Managed ACID Tables with Spark 3

Hive managed ACID tables store transactional data in base and delta files. Standard Spark Hive integration might not correctly process the transactional state of these tables, which can result in missing, duplicate, or deleted rows being returned.

Use Hive Warehouse Connector (HWC) to access Hive managed ACID tables from Spark 3.

HWC enables Spark to interact with Hive managed ACID tables while handling the required Hive transactional processing.

Prerequisites

Before you begin:

Ensure that Spark 3 is installed and configured.
Identify the ODP version installed on the cluster.
Obtain the HiveServer2 JDBC URL.
For Kerberos-enabled clusters, obtain the HiveServer2 Kerberos principal.

Start Spark Shell with HWC

Start Spark Shell and specify the HWC JAR:

spark-shell \ --master yarn \ --jars /usr/odp/<ODP_VERSION>/hive_warehouse_connector/hive-warehouse-connector-spark3-assembly-1.0.0.jar \ --conf spark.sql.hive.hiveserver2.jdbc.url="<HIVE_JDBC_URL>" \ --conf spark.sql.hive.hiveserver2.jdbc.url.principal="hive/_HOST@EXAMPLE.COM"

Replace <ODP_VERSION> and <HIVE_JDBC_URL> with the values for your environment.

Start PySpark with HWC

Start PySpark and specify the HWC JAR and Python package:

pyspark \ --master yarn \ --jars /usr/odp/<ODP_VERSION>/hive_warehouse_connector/hive-warehouse-connector-spark3-assembly-1.0.0.jar \ --py-files /usr/odp/<ODP_VERSION>/hive_warehouse_connector/pyspark3_hwc-1.0.0.zip \ --conf spark.sql.hive.hiveserver2.jdbc.url="<HIVE_JDBC_URL>" \ --conf spark.sql.hive.hiveserver2.jdbc.url.principal="hive/_HOST@EXAMPLE.COM"

Create an HWC session

Spark Shell

Import HiveWarehouseSession:

import com.acceldata.hwc.HiveWarehouseSession

Create an HWC session:

val hive = HiveWarehouseSession.session(spark).build()

Last updated on Aug 31, 2026

Was this page helpful?