Configure ODP Hive and Tez

This page describes how to enable Hive query statistics, configure JMX for HiveServer2 and Metastore, and place Pulse hook JARs for Hive and Tez so that Pulse can collect query data and metrics.

View Hive Table Details in Pulse

To display Hive table details with data in Pulse:

1. Enable automatic statistics gathering. In the hive-site.xml file, set the following properties to true:

hive.stats.autogather

hive.stats.column.autogather

This allows Hive to compute table statistics automatically.

2. Compute statistics manually (optional). You can also run the following Hive command to compute table statistics manually:

ANALYZE TABLE <table_name> COMPUTE STATISTICS;

Configure HiveServer2 JMX

In the Ambari UI:

1. Navigate to Hive > Configs > Advanced hive-env > hive-env template.
2. In hive-envtemplate``, choose and add one of the following JMX configurations based on your security requirements.

Enable JMX without Security on JMX Remote Port

To enable JMX port without any security, add the following parameters at the end of the file:

if [ "$SERVICE" = "hiveserver2" ]; then

export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=8008"

fi

Enable Basic Authentication on JMX Remote Port (Optional)

To enable basic authentication on the JMX remote port, add the following parameters:

if [ "$SERVICE" = "hiveserver2" ]; then

export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.access.file=</path/to/jmxremote.access/file> -Dcom.sun.management.jmxremote.password.file=</path/to/jmxremote.password/file> -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=8008"

fi

Enable TLS/SSL on JMX Remote Port (Optional)

To enable TLS/SSL authentication on the JMX remote port, add the following parameters:

if [ "$SERVICE" = "hiveserver2" ]; then

export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.access.file=</path/to/jmxremote.access/file> -Dcom.sun.management.jmxremote.password.file=</path/to/jmxremote.password/file> -Dcom.sun.management.jmxremote.ssl=true -Dcom.sun.management.jmxremote.registry.ssl=true -Djavax.net.ssl.keyStore=</path/to/keystore.jks/file> -Djavax.net.ssl.keyStorePassword=<Keystore Password> -Djavax.net.ssl.trustStore=</path/to/truststore.jks/file> -Djavax.net.ssl.trustStorePassword=<Truststore Password> -Dcom.sun.management.jmxremote.port=8008"

fi

Configure Hive Metastore JMX

In the Ambari UI:

1. Navigate to Hive > Configs > Advanced hive-env > hive-env template.
2. In hive-env template, choose and add one of the following JMX configurations based on your security requirements.

Enable JMX without Security on JMX Remote Port

To enable JMX port without any security, add the following parameters at the end of the file:

if [ "$SERVICE" = "metastore" ]; then

export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=8009"

fi

Enable JMX With Basic Authentication (Optional)

To enable basic authentication on the JMX remote port, add the following parameters:

if [ "$SERVICE" = "metastore" ]; then

export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.access.file=</path/to/jmxremote.access/file> -Dcom.sun.management.jmxremote.password.file=</path/to/jmxremote.password/file> -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=8008"

fi

Enable JMX With TLS/SSL Authentication (Optional)

To enable TLS/SSL authentication on the JMX remote port, add the following parameters:

if [ "$SERVICE" = "metastore" ]; then

export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.access.file=</path/to/jmxremote.access/file> -Dcom.sun.management.jmxremote.password.file=</path/to/jmxremote.password/file> -Dcom.sun.management.jmxremote.ssl=true -Dcom.sun.management.jmxremote.registry.ssl=true -Djavax.net.ssl.keyStore=</path/to/keystore.jks/file> -Djavax.net.ssl.keyStorePassword=<Keystore Password> -Djavax.net.ssl.trustStore=</path/to/truststore.jks/file> -Djavax.net.ssl.trustStorePassword=<Truststore Password> -Dcom.sun.management.jmxremote.port=8008"

fi

Place Pulse hook JARs and Set Properties (Hive and Tez)

Pulse uses Hive and Tez hooks to capture query statistics.

Hook JAR Mapping

Steps to Place Hive Hook JARs

1. Get the Hive hook JARs from Acceldata (see mapping above).
2. Place the JARs on all edge nodes, HiveServer2, and Hive interactive nodes in:

/opt/acceldata

3. Ensure the hook directory is readable and executable by all users.
4. Add the hook JARs to AUX_CLASSPATH:

• For HiveServer2, navigate to Hive > Configs > Advanced hive-env, add the following hook jar.

For example:

export AUX_CLASSPATH=/opt/acceldata/ad-hive-hook_odp 3.1.4-assembly-2.0.0.jar

• For Hive Interactive, navigate to Hive > Configs > Advanced hive-interactive-env, add the following hook jar.

For example:

export AUX_CLASSPATH=/opt/acceldata/ad-hive-hook_odp 3.1.4-assembly-2.0.0.jar

Configuration

Set the following parameters and values for Hive and Tez:

1. In the Ambari UI, go to Services > Hive.

2. On the Hive page, click the Configs tab.

3. In Configs, select Custom-hive-site and set the following:

   1. ad.events.streaming.servers=(<Pulse IP>:19009)
   2. ad.cluster=(cluster name as specified in Pulse installation)
   3. (Optional: Add this only if the YARN Optimizer is enabled) Click Add Property to add a new property, and add ad.tez.fingerprinting: true

4. Repeat the same configuration steps for Tez under Services > Tez > Configs > Custom-tez-site in the Ambari UI.

5. Save the configuration changes.

6. Restart the affected services to apply the new configuration.

Hive hook

In the Ambari UI:

1. Navigate to Hive > Configs > Advanced > General.
2. Ensure io.acceldata.hive.AdHiveHook is added (comma-separated) under:

hive.exec.failure.hooks

hive.exec.pre.hooks

hive.exec.post.hooks

Tez hook

In the Ambari UI:

1. Navigate to Tez > Configs > Advanced > General.
2. Ensure the property tez.history.logging.service.class is configured to io.acceldata.hive.AdTezEventsNatsClient.

Result

• View Hive table details in Pulse.
• HiveServer2 and Metastore expose JMX metrics.
• Pulse hooks (Hive/Tez) emit execution events to Pulse.
• Pulse hooks (Hive/Tez) send fingerprinting of applications to Pulse.