Hive

This guide covers configuring Hive to access S3A storage at the session level, enabling users to work with S3 buckets without requiring cluster-wide configuration changes or service restarts.

Configuration Methods

Method 1: Session-Level Configuration (Recommended)

This method uses secure credential files (JCEKS) stored on HDFS.

Step 1: Create a credential file on HDFS

Bash
    
xxxxxxxxxx
 
hadoop credential create fs.s3a.access.key -value <access_key> \  -provider jceks://hdfs@<namenode>:8020/user/hive/s3.jceks hadoop credential create fs.s3a.secret.key -value <secret_key> \  -provider jceks://hdfs@<namenode>:8020/user/hive/s3.jceks
Copy

Step 2: Add to `hive-site.xml`

Bash
    
 
hive.security.authorization.sqlstd.confwhitelist.append=fs\\.s3a\\.path\\.style\\.access|fs\\.s3a\\.security\\.credential\\.provider\\.path|fs\\.s3a\\.bucket\\..*\\.security\\.credential\\.provider\\.path
Copy

Step 3: Set session properties in Beeline

Bash
    
xxxxxxxxxx
 
set fs.s3a.endpoint=s3.<region>.amazonaws.com;set metaconf:fs.s3a.endpoint=s3.<region>.amazonaws.com; set fs.s3a.security.credential.provider.path=jceks://hdfs@<namenode>:8020/user/hive/s3.jceks;set metaconf:fs.s3a.security.credential.provider.path=jceks://hdfs@<namenode>:8020/user/hive/s3.jceks;
Copy

Method 2: Static Configuration (Cluster-Wide)

Add the following properties to HDFS, Hive, Tez, and Spark XML configuration files:

Bash
    
xxxxxxxxxx
 
<property>  <name>fs.s3a.access.key</name>  <value>YOUR_ACCESS_KEY</value></property> <property>  <name>fs.s3a.secret.key</name>  <value>YOUR_SECRET_KEY</value></property> <property>  <name>fs.s3a.endpoint</name>  <value>YOUR_S3_ENDPOINT</value></property> <property>  <name>fs.s3a.path.style.access</name>  <value>true</value></property>
Copy

Update hive.conf.hidden.list in Custom hiveserver2-site to exclude S3A credentials:

Bash
    
xxxxxxxxxx
 
hive.conf.hidden.list=javax.jdo.option.ConnectionPassword,\hive.server2.keystore.password,\fs.s3a.proxy.password,\dfs.adls.oauth2.credential,\fs.adl.oauth2.credential
Copy

Static configuration exposes credentials in plain text across the cluster and is not recommended for multi-tenant environments.

Known Limitations

Multiple Buckets Accessing multiple S3 buckets within a single session requires additional HMS changes and is currently not supported.
Impala Session-level S3 credential configuration is not supported.
Spark SQL INSERT S3 credentials do not propagate to HMS. Use the Spark DataFrame API instead.

Support Matrix

Supported Versions

ODP 3.2: 3.2.3.5-2 and later
ODP 3.3: 3.3.6.3-101 and later

Operation	Hive (Beeline)	Spark (Direct)	Spark + HWC
Create Table	✅	❌	✅
Insert (Static/Dynamic)	✅	⚠️*	✅
Show Partitions	✅	✅	✅
MSCK Repair	✅	❌	✅
Alter Table	✅	❌	✅
Drop Table	✅	❌	✅
Alter / Drop Partition	✅	❌	✅

Legend

✅ Supported
❌ Not Supported
⚠️* Use DataFrame API; run MSCK REPAIR in Beeline after insert

Spark + HWC Example

Bash
    
xxxxxxxxxx
 
import com.acceldata.hwc.HiveWarehouseSession val hive = HiveWarehouseSession.session(spark).build() hive.execute("set fs.s3a.endpoint=s3.<region>.amazonaws.com")hive.execute("set metaconf:fs.s3a.endpoint=s3.<region>.amazonaws.com") hive.execute("set fs.s3a.security.credential.provider.path=jceks://hdfs@<namenode>:8020/user/hive/s3.jceks")hive.execute("set metaconf:fs.s3a.security.credential.provider.path=jceks://hdfs@<namenode>:8020/user/hive/s3.jceks")
Copy

Spark DataFrame Write (Workaround for INSERT Limitation)

Bash
    
xxxxxxxxxx
 
df.write.mode("append").insertInto("table_name")
Copy

Last updated on Jan 29, 2026

Was this page helpful?

Hive

Method 1: Session-Level Configuration (Recommended)

Step 1: Create a credential file on HDFS

Step 2: Add to hive-site.xml

Step 3: Set session properties in Beeline

Method 2: Static Configuration (Cluster-Wide)

Known Limitations

Support Matrix

Spark + HWC Example

Spark DataFrame Write (Workaround for INSERT Limitation)

Step 2: Add to `hive-site.xml`