Troubleshooting ODP

This section helps you with solving and configuring common issues related to component installation in ODP.

Configure /tmp

If /tmp is required to be mounted as exec but compliance issues necessitate mounting it as noexec, follow the steps below on every node to create an alternative tmp directory:

1. Create an alternative tmp directory:

sudo mkdir -p /hadoop/tmp

sudo chown root:hadoop /hadoop/tmp

sudo chmod 1770 /hadoop/tmp

2. Update Ambari configuration for respective services:

• For HBase, YARN-HBase (Timeline Reader), and Ambari Metrics (AMS): In Ambari, update the YARN configurations yarn-hbase-env, AMS ams-hbase-env, and HBase hbase-env by adding:

export HBASE_OPTS="$HBASE_OPTS -Dorg.apache.hbase.thirdparty.io.netty.native.workdir=/hadoop/tmp"

• For Knox: Directly update gateway.sh and knoxcli.sh in the file system on all Knox hosts by appending:

-Djava.io.tmpdir=/hadoop/tmp -D*jna*.tmpdir=/hadoop/tmp

• For HDFS/YARN/MR: Override HADOOP_OPTS and append:

-Djava.io.tmpdir=/hadoop/tmp

The above adjustments ensure compatibility and functionality despite the noexec mount constraint.

Cruise Control

If Cruise Control fails to start after enabling SSL, verify the value of the following properties.

Path: In the Ambari UI > Kafka > Configs > Advanced cruise-control.

zookeeper.connect = <zk-connect-string>

zookeeper.security.enabled = true

Impala

1. Update/add these configurations respectively for smooth Impala installation

Pre-installation > Update in impala-env, if not set :

is_coordinator = true

is_executor = true

Post-installation > Add custom properties in hadoop core-site :

hadoop.proxyuser.impala.groups=*

hadoop.proxyuser.impala.hosts=*

Ranger

1. If the KMS ranger policy fails to create. Perform the following steps:

• Remove the empty KMS policy folder from the CLI
• Restart Ranger from Ambari UI
• Then, Restart Ranger KMS from Ambari UI

2. If service repo creation fails with a missing RangerLookup user, create a RangerLookup user manually from the Ranger UI.
3. On enabling Ranger SSL, if the plugin component fails to start with the following error, add htrace-core4-4.1.0-incubating.jar to the respective Ranger plugin lib path and restart the service.

Error:

Execute[(u'/usr/odp/3.2.3.5-3/ranger-ozone-plugin/ranger_credential_helper.py', '-l', u'/usr/odp/3.2.3.5-3/ranger-ozone-plugin/install/lib/*', '-f', '/etc/ranger/rl8F32342_ozone/cred.jceks', '-k', 'sslTrustStore', '-v', [PROTECTED], '-c', '1')] {'logoutput': True, 'environment': {'JAVA_HOME': u'/usr/lib/jvm/java-1.8.0-openjdk'}, 'sudo': True}

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/htrace/core/Tracer$Builder

Example solution:

cp /usr/odp/3.2.3.5-3/ranger-admin/cred/lib/htrace-core4-4.1.0-incubating.jar /usr/odp/3.2.3.5-3/ranger-ozone-plugin/install/lib/

Spark2

MySQL 8.x

• If service setup or start fails with the error "Host 'host' is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts'", follow these steps:

# Check and increase max connections

# MySQL shell

SET GLOBAL max_connect_errors=100000;

• If MySQL connection fails with the error "SQLException: SQL state: 08001 java.sql.SQLNonTransientConnectionException: Public Key Retrieval is not allowed ErrorCode: 0", this is caused by the default authentication plugin change from mysql_native_password to caching_sha2_password in MySQL 8.0. See MySQL 8.0 Reference Manual :: 3.5 Changes in MySQL 8.0.

To fix this issue, add ?allowPublicKeyRetrieval=true&useSSL=false to the MySQL JDBC connection string. For example: "jdbc:mysql://hostname:3306/ranger?allowPublicKeyRetrieval=true&useSSL=false".