Hive

This guide covers configuring Hive to access S3A storage at the session level, enabling users to work with S3 buckets without requiring cluster-wide configuration changes or service restarts.

Configuration Methods

Method 1: Session-Level Configuration (Recommended)

This method uses secure credential files (JCEKS) stored on HDFS.

Step 1: Create a credential file on HDFS

hadoop credential create fs.s3a.access.key -value <access_key> \ -provider jceks://hdfs@<namenode>:8020/user/hive/s3.jceks hadoop credential create fs.s3a.secret.key -value <secret_key> \ -provider jceks://hdfs@<namenode>:8020/user/hive/s3.jceks

Step 2: Add to `hive-site.xml`

hive.security.authorization.sqlstd.confwhitelist.append=fs\\.s3a\\.path\\.style\\.access|fs\\.s3a\\.security\\.credential\\.provider\\.path|fs\\.s3a\\.bucket\\..*\\.security\\.credential\\.provider\\.path

Step 3: Set session properties in Beeline

set fs.s3a.endpoint=s3.<region>.amazonaws.com; set metaconf:fs.s3a.endpoint=s3.<region>.amazonaws.com; set fs.s3a.security.credential.provider.path=jceks://hdfs@<namenode>:8020/user/hive/s3.jceks; set metaconf:fs.s3a.security.credential.provider.path=jceks://hdfs@<namenode>:8020/user/hive/s3.jceks;

Method 2: Static Configuration (Cluster-Wide)

Add the following properties to HDFS, Hive, Tez, and Spark XML configuration files:

<property> <name>fs.s3a.access.key</name> <value>YOUR_ACCESS_KEY</value> </property> <property> <name>fs.s3a.secret.key</name> <value>YOUR_SECRET_KEY</value> </property> <property> <name>fs.s3a.endpoint</name> <value>YOUR_S3_ENDPOINT</value> </property> <property> <name>fs.s3a.path.style.access</name> <value>true</value> </property>

Update hive.conf.hidden.list in Custom hiveserver2-site to exclude S3A credentials:

hive.conf.hidden.list=javax.jdo.option.ConnectionPassword,\ hive.server2.keystore.password,\ fs.s3a.proxy.password,\ dfs.adls.oauth2.credential,\ fs.adl.oauth2.credential

Warning

Static configuration exposes credentials in plain text across the cluster and is not recommended for multi-tenant environments.

Known Limitations

Multiple Buckets Accessing multiple S3 buckets within a single session requires additional HMS changes and is currently not supported.
Impala Session-level S3 credential configuration is not supported.
Spark SQL INSERT S3 credentials do not propagate to HMS. Use the Spark DataFrame API instead.

Support Matrix

Supported Versions

ODP 3.2: 3.2.3.5-2, 3.2.3.5-3,and later
ODP 3.3: 3.3.6.3-101 and later

Operation	Hive (Beeline)	Spark (Direct)	Spark + HWC
Create Table	✅	❌	✅
Insert (Static/Dynamic)	✅	⚠️*	✅
Show Partitions	✅	✅	✅
MSCK Repair	✅	❌	✅
Alter Table	✅	❌	✅
Drop Table	✅	❌	✅
Alter / Drop Partition	✅	❌	✅

Legend

✅ Supported
❌ Not Supported
⚠️* Use DataFrame API; run MSCK REPAIR in Beeline after insert

Spark + HWC Example

import com.acceldata.hwc.HiveWarehouseSession val hive = HiveWarehouseSession.session(spark).build() hive.execute("set fs.s3a.endpoint=s3.<region>.amazonaws.com") hive.execute("set metaconf:fs.s3a.endpoint=s3.<region>.amazonaws.com") hive.execute("set fs.s3a.security.credential.provider.path=jceks://hdfs@<namenode>:8020/user/hive/s3.jceks") hive.execute("set metaconf:fs.s3a.security.credential.provider.path=jceks://hdfs@<namenode>:8020/user/hive/s3.jceks")

Spark DataFrame Write (Workaround for INSERT Limitation)

df.write.mode("append").insertInto("table_name")

Last updated on Apr 29, 2026

Was this page helpful?