Configure HDP Hive, Tez, and Sqoop

This page describes how to configure Hive, Tez, and Sqoop services in HDP clusters to enable Pulse to collect query statistics, performance metrics, and lineage information.

Configure HDP Hive for Pulse

This section describes how to configure Hive so that Pulse can collect query data and metrics.

Configure HiveServer2 JMX

In the Ambari UI:

  1. Navigate to Hive > Configs > Advanced hive-env > hive-env template.

  2. In hive-envtemplate``, choose and add one of the following JMX configurations based on your security requirements.

Enable JMX without Security on JMX Remote Port

To enable JMX port without any security, add the following parameters at the end of the file:

if [ "$SERVICE" = "hiveserver2" ]; then export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=8008" fi
Warning

Avoid JMX Changes for Hive 1.x Using the MR Engine

Do not enable JMX for Hive 1.x when using the MapReduce (MR) execution engine. A known bug can cause query failures if JMX is enabled.

Enable Basic Authentication on JMX Remote Port (Optional)

To enable basic authentication on the JMX remote port, add the following parameters:

if [ "$SERVICE" = "hiveserver2" ]; then export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.access.file=</path/to/jmxremote.access/file> -Dcom.sun.management.jmxremote.password.file=</path/to/jmxremote.password/file> -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=8008" fi

Enable TLS/SSL on JMX Remote Port (Optional)

To enable TLS/SSL authentication on the JMX remote port, add the following parameters:

if [ "$SERVICE" = "hiveserver2" ]; then export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.access.file=</path/to/jmxremote.access/file> -Dcom.sun.management.jmxremote.password.file=</path/to/jmxremote.password/file> -Dcom.sun.management.jmxremote.ssl=true -Dcom.sun.management.jmxremote.registry.ssl=true -Djavax.net.ssl.keyStore=</path/to/keystore.jks/file> -Djavax.net.ssl.keyStorePassword=<Keystore Password> -Djavax.net.ssl.trustStore=</path/to/truststore.jks/file> -Djavax.net.ssl.trustStorePassword=<Truststore Password> -Dcom.sun.management.jmxremote.port=8008" fi

Configure Hive Metastore JMX

In the Ambari UI:

  1. Navigate to Hive > Configs > Advanced hive-env > hive-env template.

  2. In hive-envtemplate``, choose and add one of the following JMX configurations based on your security requirements.

Enable JMX without Security on JMX Remote Port

To enable JMX port without any security, add the following parameters at the end of the file:

if [ "$SERVICE" = "metastore" ]; then export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=8009" fi

Enable Basic Authentication on JMX Remote Port (Optional)

To enable basic authentication on the JMX remote port, add the following parameters:

if [ "$SERVICE" = "metastore" ]; then export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.access.file=</path/to/jmxremote.access/file> -Dcom.sun.management.jmxremote.password.file=</path/to/jmxremote.password/file> -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=8008" fi

Enable TLS/SSL on JMX Remote Port (Optional)

To enable TLS/SSL authentication on the JMX remote port, add the following parameters:

if [ "$SERVICE" = "metastore" ]; then export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.access.file=</path/to/jmxremote.access/file> -Dcom.sun.management.jmxremote.password.file=</path/to/jmxremote.password/file> -Dcom.sun.management.jmxremote.ssl=true -Dcom.sun.management.jmxremote.registry.ssl=true -Djavax.net.ssl.keyStore=</path/to/keystore.jks/file> -Djavax.net.ssl.keyStorePassword=<Keystore Password> -Djavax.net.ssl.trustStore=</path/to/truststore.jks/file> -Djavax.net.ssl.trustStorePassword=<Truststore Password> -Dcom.sun.management.jmxremote.port=8008" fi

Place Hive Hook JARs

Pulse uses Hive hooks to capture query statistics.

Hook JAR Mapping

Distro Version

Hive Version

Tez Version

Pulse Hook Jar Name

HDP 2.x

1.2.x

0.7.x

ad-hive-hook__hdp__1.2.x-assembly-x.y.x.jar

HDP 2.x

2.1.x (LLAP)

0.7.x

ad-hive-hook__hdp__2.1.x-assembly-x.y.x.jar

HDP 3.1.0.x

3.1.x

0.9.x

ad-hive-hook__hdp__3.1.0.3.1.0.0-78-assembly-x.y.x.jar

HDP 3.1.4.x

3.1.x

0.9.x

ad-hive-hook__hdp__3.1.0.3.1.4.0-315-assembly-x.y.z.jar

Info

X.Y.Z >> For details about supported hook versions on each Pulse version, see Supported Hook Versions.

  • For Pulse 4.0.x, use the hook version 2.0.1

  • For Pulse 4.1.x, use the hook version 2.1.2

Steps to Place Hive Hook JARs

  1. Get the Hive hook JARs from Acceldata (see mapping below).

  2. Remove the existing Hive/Tez hook JAR from /opt/acceldata/ before placing the new JAR.

  3. Place the JARs on HiveServer2 and Hive interactive nodes in:

/opt/acceldata

  1. Ensure the hook directory is readable and executable by all users.

  2. Add the hook JARs to AUX_CLASSPATH:

Info

Update the hook JAR name in the following properties to match your installed HDP distribution version.

  • For HiveServer2, navigate to Hive > Configs > Advanced hive-env, add the following hook jar.

For example:

export AUX_CLASSPATH=/opt/acceldata/<JARNAME>

<JARNAME>: Select the corresponding Hook JAR from the mapping table based on your Hive and Tez versions.

  • For Hive Interactive, navigate to Hive > Configs > Advanced hive-interactive-env, add the following hook jar.

For example:

export AUX_CLASSPATH=/opt/acceldata/<JARNAME>

<JARNAME>: Select the corresponding Hook JAR from the mapping table based on your Hive and Tez versions.

Info

X.Y.Z >> For details about supported hook versions on each Pulse version, see Supported Hook Versions.

  • For Pulse 4.0.x, use the hook version 2.0.1.

  • For Pulse 4.1.x, use the hook version 2.1.2

  1. Add the Pulse configurations in Hive > Configs > Custom hive-site and Custom hive-interactive-site.

    1. ad.events.streaming.servers=(<Pulse IP>:19009)

    2. ad.cluster=(cluster name as specified in Pulse installation)

  2. Add io.acceldata.hive.AdHiveHook (use a comma if needed) to the following properties in Hive > Configs > General.

hive.exec.failure.hooks hive.exec.pre.hooks hive.exec.post.hooks

Configure HDP Tez for Pulse

This section describes how to configure Tez so that Pulse can collect query events.

Prerequisites

  • Obtain the Hive hook JARs from the Acceldata team.

  • Review the mapping table to identify the correct JAR version for your HDP distribution. For details, see the section Hook JAR Mapping on this page.

Steps for Configuration

  1. Identify the hook JAR location.

  • For Hive 3.x on HDP 3.x (example: HDP 3.1.4):

Use the JAR ad-hive-hook_hdp_3.1.0.3.1.4.0-315-assembly-x.y.z.jar on the HDFS path:

/hdp/apps/${hdp.version}/tez/tez.tar.gz

  • For Hive 1.x on HDP 2.6.x: Use the JAR ad-hive-hook_hdp_1.2.x-assembly-x.y.z.jar on the HDFS path.

$plugin[89wocz3mp33]

  • For Hive 2.x (LLAP) on HDP 2.6.x:

Use the JAR ad-hive-hook_hdp_2.1.x-assembly-x.y.z.jar on the HDFS path:

/hdp/apps/${hdp.version}/tez_hive2/tez.tar.gz
Info

X.Y.Z >> For details about supported hook versions on each Pulse version, see Supported Hook Versions.

  1. Update the Tez tarball with the Pulse hook JAR.

Info

During upgrades or hook updates, ensure the Tez tarball contains only the latest Acceldata hook JAR. Remove any older hook files to avoid conflicts.

# Create a directory mkdir -p tez_pack/ && cd tez_pack # Take backup of existing tez tarball in HDFS /tmp hdfs dfs -cp /hdp/apps/<cluster_version>/tez/tez.tar.gz /tmp # Download tez tarball from HDFS to local, switch to accesible user hdfs dfs -get /hdp/apps/<cluster_version>/tez/tez.tar.gz . # Unpack the tarball tar -zxvf tez.tar.gz # Remove existing Pulse hook jar from tez libs/ (if exists) rm ./lib/<existing_pulse_hook.jar> # Copy Pulse hook jar to tez libs/ cp </location../../pulse_hook.jar> ./lib/ # Package tez tarball tar -cvzf /tmp/tez.tar.gz . # Upload back and provide right permissions and ownership hdfs dfs -put -f /tmp/tez.tar.gz /hdp/apps/<cluster_version>/tez/tez.tar.gz hdfs dfs -chown hdfs:hadoop /hdp/apps/<cluster_version>/tez/tez.tar.gz hdfs dfs -chmod 755 /hdp/apps/<cluster_version>/tez/tez.tar.gz

  1. Update the tez-site properties.

    • In the Ambari UI, navigate to: Tez > Configs > Custom tez-site.

    • Add or update the following properties:

tez.history.logging.service.class=io.acceldata.hive.AdTezEventsNatsClient ad.events.streaming.servers (PULSE_IP:19009) ad.cluster (your cluster name, ex: ad_hdp3_dev)

  • Optional (Hive 3.x only): The property ad.hdfs.sink is set to true by default. If set to false, Tez will not publish query metadata proto logs to HDFS.

Configure HDP Sqoop for Pulse

This section describes how to configure Sqoop so that Pulse can collect query data.

  1. Copy and place the Hive hook JARs (from the table above) into the Sqoop classpath directory:

Example path:

/usr/hdp/current/sqoop-client/lib

  1. For clusters with LLAP (Hive Interactive) enabled, place both Hive 1.2.x and Hive 2.1.x hook JARs in the classpath.

Result

  • Hive exposes metrics over the configured port.

  • Hive, Tez, and Sqoop are configured with JMX monitoring, hook JARs, and Pulse integration for query and job statistics.


Next to read:
Configure HDP Spark

For additional help, contact www.acceldata.force.com OR call our service desk +1 844 9433282

Copyright © 2026

On This Page
Configure HDP Hive, Tez, and SqoopConfigure HDP Hive for PulseConfigure HDP Tez for PulseConfigure HDP Sqoop for PulseResultConfigure HiveServer2 JMXConfigure Hive Metastore JMXPlace Hive Hook JARsPrerequisitesSteps for ConfigurationEnable JMX without Security on JMX Remote PortEnable Basic Authentication on JMX Remote Port (Optional)Enable TLS/SSL on JMX Remote Port (Optional)Enable JMX without Security on JMX Remote PortEnable Basic Authentication on JMX Remote Port (Optional)Enable TLS/SSL on JMX Remote Port (Optional)Hook JAR MappingSteps to Place Hive Hook JARs