Configure ODP Hive and Tez

This page describes how to enable Hive query statistics, configure JMX for HiveServer2 and Metastore, and place Pulse hook JARs for Hive and Tez so that Pulse can collect query data and metrics.

View Hive Table Details in Pulse

To display Hive table details with data in Pulse:

1. Enable automatic statistics gathering. In the hive-site.xml file, set the following properties to true:

hive.stats.autogather

hive.stats.column.autogather

This allows Hive to compute table statistics automatically.

2. Compute statistics manually (optional). You can also run the following Hive command to compute table statistics manually:

ANALYZE TABLE <table_name> COMPUTE STATISTICS;

Configure HiveServer2 JMX

In the Ambari UI:

1. Navigate to Hive > Configs > Advanced hive-env > hive-env template.
2. In hive-envtemplate``, choose and add one of the following JMX configurations based on your security requirements.

Enable JMX without Security on JMX Remote Port

To enable JMX port without any security, add the following parameters at the end of the file:

if [ "$SERVICE" = "hiveserver2" ]; then

export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=8008"

fi

Enable Basic Authentication on JMX Remote Port (Optional)

To enable basic authentication on the JMX remote port, add the following parameters:

if [ "$SERVICE" = "hiveserver2" ]; then

export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.access.file=</path/to/jmxremote.access/file> -Dcom.sun.management.jmxremote.password.file=</path/to/jmxremote.password/file> -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=8008"

fi

Enable TLS/SSL on JMX Remote Port (Optional)

To enable TLS/SSL authentication on the JMX remote port, add the following parameters:

if [ "$SERVICE" = "hiveserver2" ]; then

export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.access.file=</path/to/jmxremote.access/file> -Dcom.sun.management.jmxremote.password.file=</path/to/jmxremote.password/file> -Dcom.sun.management.jmxremote.ssl=true -Dcom.sun.management.jmxremote.registry.ssl=true -Djavax.net.ssl.keyStore=</path/to/keystore.jks/file> -Djavax.net.ssl.keyStorePassword=<Keystore Password> -Djavax.net.ssl.trustStore=</path/to/truststore.jks/file> -Djavax.net.ssl.trustStorePassword=<Truststore Password> -Dcom.sun.management.jmxremote.port=8008"

fi

Configure Hive Metastore JMX

In the Ambari UI:

1. Navigate to Hive > Configs > Advanced hive-env > hive-env template.
2. In hive-env template, choose and add one of the following JMX configurations based on your security requirements.

Enable JMX without Security on JMX Remote Port

To enable JMX port without any security, add the following parameters at the end of the file:

if [ "$SERVICE" = "metastore" ]; then

export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=8009"

fi

Enable JMX With Basic Authentication (Optional)

To enable basic authentication on the JMX remote port, add the following parameters:

if [ "$SERVICE" = "metastore" ]; then

export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.access.file=</path/to/jmxremote.access/file> -Dcom.sun.management.jmxremote.password.file=</path/to/jmxremote.password/file> -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=8008"

fi

Enable JMX With TLS/SSL Authentication (Optional)

To enable TLS/SSL authentication on the JMX remote port, add the following parameters:

if [ "$SERVICE" = "metastore" ]; then

export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.access.file=</path/to/jmxremote.access/file> -Dcom.sun.management.jmxremote.password.file=</path/to/jmxremote.password/file> -Dcom.sun.management.jmxremote.ssl=true -Dcom.sun.management.jmxremote.registry.ssl=true -Djavax.net.ssl.keyStore=</path/to/keystore.jks/file> -Djavax.net.ssl.keyStorePassword=<Keystore Password> -Djavax.net.ssl.trustStore=</path/to/truststore.jks/file> -Djavax.net.ssl.trustStorePassword=<Truststore Password> -Dcom.sun.management.jmxremote.port=8008"

fi

Place Pulse hook JARs and Set Properties (Hive and Tez)

Pulse uses Hive and Tez hooks to capture query statistics.

Hook JAR Mapping

Steps to Place Hive Hook JARs

1. Get the Hive hook JARs from Acceldata (see mapping above).
2. Place the JARs on all edge nodes, HiveServer2, and Hive interactive nodes in:

/opt/acceldata

3. Ensure the hook directory is readable and executable by all users.
4. Add the hook JARs to AUX_CLASSPATH:

• For HiveServer2, navigate to Hive > Configs > Advanced hive-env, add the following hook jar.

For example:

export AUX_CLASSPATH=/opt/acceldata/ad-hive-hook_odp 3.1.4-assembly-x.y.x.jar

• For Hive Interactive, navigate to Hive > Configs > Advanced hive-interactive-env, add the following hook jar.

For example:

export AUX_CLASSPATH=/opt/acceldata/ad-hive-hook_odp 3.1.4-assembly-x.y.x.jar

Configuration

Set the following parameters and values for Hive and Tez:

1. In the Ambari UI, go to Services > Hive.

2. On the Hive page, click the Configs tab.

3. In Configs, select Custom-hive-site and set the following:

   1. ad.events.streaming.servers=(<Pulse IP>:19009)
   2. ad.cluster=(cluster name as specified in Pulse installation)
   3. (Optional: Add this only if the YARN Optimizer is enabled) Click Add Property to add a new property, and add ad.tez.fingerprinting: true

4. Repeat the same configuration steps for Tez under Services > Tez > Configs > Custom-tez-site in the Ambari UI.

5. Save the configuration changes.

6. Restart the affected services to apply the new configuration.

Hive hook

In the Ambari UI:

1. Navigate to Hive > Configs > Advanced > General.
2. Ensure io.acceldata.hive.AdHiveHook is added (comma-separated) under:

hive.exec.failure.hooks

hive.exec.pre.hooks

hive.exec.post.hooks

Tez hook

In the Ambari UI:

1. Navigate to Tez > Configs > Advanced > General.
2. Ensure the property tez.history.logging.service.class is configured to io.acceldata.hive.AdTezEventsNatsClient.

Configure ODP Tez for Pulse

This section describes how to configure Tez so that Pulse can collect query events.

Prerequisites

• Obtain the Hive hook JARs from the Acceldata team.
• Review the mapping table to identify the correct JAR version for your HDP distribution. For details, see the section Hook JAR Mapping on this page.

Steps for Configuration

1. Identify the hook JAR location.

For example:

• For Hive 3.1.4 on ODP:

Use the JAR ad-hive-hook_odp_3.1.4-assembly-X.Y.Z.jar on the HDFS path:

/odp/apps/${odp.version}/tez/tez.tar.gz

For details on the supported hooks for different Hive and Tez versions, see Place Pulse hook JARs and Set Properties (Hive and Tez)

2. Update the Tez tarball with the Pulse hook JAR.

# Create a directory

mkdir -p tez_pack/ && cd tez_pack

​

# Take backup of existing tez tarball in HDFS /tmp

hdfs dfs -cp /odp/apps/<cluster_version>/tez/tez.tar.gz /tmp

​

# Download tez tarball from HDFS to local, switch to accesible user

hdfs dfs -get /odp/apps/<cluster_version>/tez/tez.tar.gz .

​

# Unpack the tarball

tar -zxvf tez.tar.gz

​

# Copy Pulse hook jar to tez libs/

cp </location../../pulse_hook.jar> ./lib/

​

# Package tez tarball

tar -cvzf /tmp/tez.tar.gz .

​

# Upload back and provide right permissions and ownership

hdfs dfs -put -f /tmp/tez.tar.gz /odp/apps/<cluster_version>/tez/tez.tar.gz

hdfs dfs -chown hdfs:hadoop /odp/apps/<cluster_version>/tez/tez.tar.gz

hdfs dfs -chmod 755 /odp/apps/<cluster_version>/tez/tez.tar.gz

3. Update the tez-site properties.
   • In the Ambari UI, navigate to: Tez > Configs > Custom tez-site.
   • Add or update the following properties:

tez.history.logging.service.class=io.acceldata.hive.AdTezEventsNatsClient

ad.events.streaming.servers (PULSE_IP:19009)

ad.cluster (your cluster name, ex: ad_odp_dev)

• Optional (Hive 3.x only): The property ad.hdfs.sink is set to true by default. If set to false, Tez will not publish query metadata proto logs to HDFS.

Result

• View Hive table details in Pulse.
• HiveServer2 and Metastore expose JMX metrics.
• Pulse hooks (Hive/Tez) emit execution events to Pulse.
• Pulse hooks (Hive/Tez) send fingerprinting of applications to Pulse.