Known Limitations

Ranger

Issue description: Component generates warning logs

The RangerGdsInfoRefresher component generates warning logs, as it is a work-in-progress feature still under active development in the Apache Ranger community.

Workaround

To suppress these logs, update the Ranger properties in the respective service.

For example, to suppress the RangerGdsInfoRefresher logs in Kafka, add the following property to Ambari UI > Kafka > ranger-kafka-plugin-properties.

ranger.plugin.kafka.policyengine.option.disable.gdsinfo.retriever=true

Proceed with the Ranger service restart from the Ambari UI to apply the changes. It must start smoothly.

Hive

Issue description: Hive Compaction Failure

Hive compaction jobs fail due to a version conflict with the protobuf-java library.

• This typically occurs during a minor or major compaction operation on ORC-backed Hive tables.
• This can be resolved by updating MapReduce Classpath.
• The issue is identified in ODP versions 3.3.6.0-1 and 3.3.6.1-1.
• The issue will be fixed in ODP version 3.3.6.2-1.

When executing the compaction command, the jobs fail with the following error message in the logs.

ALTER TABLE employee COMPACT 'minor';

16:06:50.650 [main] ERROR org.apache.hadoop.mapred.YarnChild - Error running child : java.lang.NoSuchMethodError: 'com.google.protobuf.LazyStringList com.google.protobuf.LazyStringList.getUnmodifiableView()'
 at org.apache.orc.OrcProto$Type$Builder.buildPartial(OrcProto.java:20430)
 at org.apache.orc.OrcProto$Type$Builder.build(OrcProto.java:20408)
 at org.apache.orc.OrcUtils.appendOrcTypes(OrcUtils.java:203)
 at org.apache.orc.OrcUtils.getOrcTypes(OrcUtils.java:110)
 at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.<init>(OrcRawRecordMerger.java:1031)
 at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:2459)
 at org.apache.hadoop.hive.ql.txn.compactor.MRCompactor$CompactorMap.map(MRCompactor.java:823)
 at org.apache.hadoop.hive.ql.txn.compactor.MRCompactor$CompactorMap.map(MRCompactor.java:799)
 at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
 at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:466)
 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:350)
 at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178)
 at java.base/java.security.AccessController.doPrivileged(Native Method)
 at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
 at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172)

This indicates a version mismatch in the protobuf-java library used during MapReduce execution, resulting in a NoSuchMethodError.

Root Cause

The Hive client uses a newer version of the protobuf-java library that includes the getUnmodifiableView() method. However, during compaction, the MapReduce job loads an older version of protobuf-java from the cluster's classpath, which lacks this method, leading to a runtime error.

Workaround

To ensure the correct protobuf-java version is used during MapReduce jobs, you need to explicitly update the MapReduce classpath to include the Hive client’s protobuf-java jar before other entries.

The steps to fix via Ambari are as follows:

Pre-requisites: Ensure that the Tez client is installed on all NodeManagers.

• Log into the Ambari UI.
• Navigate to MapReduce2 → Configs → Advanced → Advanced mapred-site.
• Locate the property mapreduce.application.classpath.
• Prepend the following path to the existing value (do not overwrite the current classpath):

/usr/odp/current/tez-client/lib/protobuf-java-3.21.1.jar:

Also, ensure to confirm if the same version exists on the cluster, if not, use the available version.

The final value must look similar to the following:

/usr/odp/current/tez-client/lib/protobuf-java-3.21.1.jar:$PWD/mr-framework/hadoop/share... (rest of the classpath)

• Save the changes.
• Restart the necessary services (typically MapReduce and Hive components) to apply the new configuration.

Verify

Use the following steps to verify the fix.

• Re-run the following command.

ALTER TABLE employee COMPACT 'minor';

• Monitor the YARN application logs for successful completion.
• Confirm that the error is no longer present and the compaction completes as expected.