Use Hive Warehouse Connector with Spark 4

Hive Warehouse Connector (HWC) enables Spark 4 applications to access and operate on Hive tables, including Hive managed transactional tables.

This page describes how to configure HWC with Spark 4.1.1, connect to HiveServer2, and perform basic Hive operations.

Prerequisites

Before you begin, ensure that your environment meets the following requirements:

Component	Version
ODP	3.3.6.4-1
Spark	4.1.1.3.3.6.4-1
Scala	2.13.17
Java	OpenJDK 17.0.19

HWC JAR

The HWC JAR for Spark 4 is available at:

/usr/odp/3.3.6.4-1/hive_warehouse_connector_spark4/hive-warehouse-connector-spark4-assembly-1.0.0.jar

Authenticate with Kerberos

On a Kerberos-enabled cluster, obtain a Kerberos ticket before you start Spark Shell.

Run:

kinit -kt /etc/security/keytabs/spark.headless.keytab \ spark-rl8iter1@ADSRE.COM

Replace the principal with the Spark principal for your environment.

Verify the Kerberos ticket:

klist

Verify that the output contains a valid ticket for the Spark principal.

Start Spark Shell with HWC

Set SPARK_MAJOR_VERSION to 4 and start Spark Shell with the required HWC configurations:

SPARK_MAJOR_VERSION=4 spark-shell \ --conf spark.sql.hive.hiveserver2.jdbc.url.principal=hive/_HOST@ADSRE.COM \ --conf spark.sql.extensions=com.acceldata.spark.sql.rule.Extensions \ --conf spark.datasource.hive.warehouse.read.mode=JDBC_CLUSTER \ --conf spark.kryo.registrator=com.qubole.spark.hiveacid.util.HiveAcidKyroRegistrator \ --conf spark.sql.hive.hiveserver2.jdbc.url="jdbc:hive2://rl8-27.acceldata.ce:2181,rl8-26.acceldata.ce:2181,rl8-28.acceldata.ce:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2" \ --jars /usr/odp/3.3.6.4-1/hive_warehouse_connector_spark4/hive-warehouse-connector-spark4-assembly-1.0.0.jar

Replace the HiveServer2 JDBC URL and Kerberos principal with the values for your environment.

HWC configuration

Configuration	Value	Description
`SPARK_MAJOR_VERSION`	`4`	Uses the Spark 4 binaries.
`spark.sql.hive.hiveserver2.jdbc.url.principal`	HiveServer2 Kerberos principal	Specifies the Kerberos principal for HiveServer2.
`spark.sql.extensions`	`com.acceldata.spark.sql.rule.Extensions`	Loads the required SQL extensions.
`spark.datasource.hive.warehouse.read.mode`	`JDBC_CLUSTER`	Configures executors to read data through JDBC.
`spark.kryo.registrator`	`com.qubole.spark.hiveacid.util.HiveAcidKyroRegistrator`	Configures the Kryo registrator required for Hive ACID tables.
`spark.sql.hive.hiveserver2.jdbc.url`	HiveServer2 JDBC URL	Specifies the HiveServer2 connection.

After Spark Shell starts, verify that it uses Spark 4.1.1, Scala 2.13, and Java 17.

Create a Hive Warehouse session

Import the required HWC classes:

import com.acceldata.hwc.HiveWarehouseSession import com.acceldata.hwc.HiveWarehouseSession._

Create the Hive Warehouse session:

val hive = HiveWarehouseSession.session(spark).build()

Use the hive session to perform Hive operations.

List databases

Run:

hive.showDatabases().show()

Verify that the expected Hive databases are displayed.

You can also list databases by using executeQuery:

hive.executeQuery("SHOW DATABASES").show()

List tables

Run:

hive.showTables().show()

To select a database and list its tables, run:

hive.executeQuery("USE default").show() hive.executeQuery("SHOW TABLES").show()

Replace default with the required Hive database.

View a table schema

To view the columns and data types for a table, run:

hive.executeQuery("DESC employeedata").show()

Replace employeedata with the required table name.

Create a table

The following example creates an ORC table:

hive.executeQuery( "CREATE TABLE IF NOT EXISTS spark4_test " + "(id INT, name STRING) STORED AS ORC" ).show()

Note: Use the ORC/ACID format for HWC write operations.

Insert data

Insert data into the table:

hive.executeQuery( "INSERT INTO spark4_test VALUES " + "(1, 'Alice'), (2, 'Bob'), (3, 'Charlie')" ).show()

Read table data

Query the table:

hive.executeQuery( "SELECT * FROM spark4_test" ).show()

To verify the number of rows, run:

hive.executeQuery( "SELECT COUNT(*) AS total FROM spark4_test" ).show()

Troubleshoot HWC

Multiple values for the principal

You might receive an error indicating that multiple values are configured for the principal property.

This issue occurs when you specify the HiveServer2 Kerberos principal in both:

spark.sql.hive.hiveserver2.jdbc.url.principal

and the JDBC URL:

principal=<HIVE_PRINCIPAL>

Specify the principal in only one location.

We recommend using:

spark.sql.hive.hiveserver2.jdbc.url.principal

and omitting principal from the JDBC URL.

Unsupported mechanism type PLAIN

The following error can occur when executors attempt to establish a non-Kerberos SASL connection to HiveServer2:

Unsupported mechanism type PLAIN

Verify that the HiveServer2 principal is configured:

spark.sql.hive.hiveserver2.jdbc.url.principal

Alternatively, specify the principal in the HiveServer2 JDBC URL.

Important: Don't configure the principal in both locations.

GSS initiate failed

The following error can occur when executors don't have valid Kerberos credentials to establish a GSSAPI connection with HiveServer2:

GSS initiate failed

This issue can occur with JDBC_CLUSTER mode because executors establish their own JDBC connections.

To resolve the issue, obtain a valid Kerberos ticket before you start Spark Shell:

kinit -kt /etc/security/keytabs/spark.headless.keytab \ spark-rl8iter1@ADSRE.COM

Alternatively, configure HWC to use DIRECT_READER_V2:

--conf spark.datasource.hive.warehouse.read.mode=DIRECT_READER_V2

With DIRECT_READER_V2, executors read data from HDFS by using delegation tokens instead of establishing JDBC connections.

Last updated on Aug 31, 2026

Was this page helpful?