Tez and MapReduce Service Checks Fail After ODP Upgrade

After upgrading an ODP cluster, Tez and MapReduce service checks may fail with errors indicating that YARN cannot download required application resources from HDFS.

This issue typically occurs when the Tez or MapReduce application archives stored in HDFS are missing, corrupted, or have a file size of zero bytes.

Symptoms

Tez Service Check Failure

xxxxxxxxxx

Failed to download resource

hdfs:///odp/apps/<version>/tez/tez.tar.gz

org.apache.hadoop.yarn.exceptions.YarnException:

Download and unpack failed

MapReduce Service Check Failure

xxxxxxxxxx

Failed to download resource

hdfs:///odp/apps/<version>/mapreduce/mapreduce.tar.gz

org.apache.hadoop.yarn.exceptions.YarnException:

Download and unpack failed

Applications may repeatedly fail with:

xxxxxxxxxx

AM Container exited with exitCode: -1000

Cause

Tez and MapReduce rely on application archives stored in HDFS under:

xxxxxxxxxx

/odp/apps/<version>/tez/

/odp/apps/<version>/mapreduce/

During service startup or service checks, YARN distributes these archives to containers.

If the archive:

• Does not exist
• Is corrupted
• Has a size of 0 bytes

YARN cannot unpack the application resources and the service check fails.

Verify the Issue

Check the file sizes:

xxxxxxxxxx

hdfs dfs -ls /odp/apps/<version>/tez

hdfs dfs -ls /odp/apps/<version>/mapreduce

Example of a problematic archive:

xxxxxxxxxx

-rw-r--r-- 0 hdfs hdfs 0 tez.tar.gz

-rw-r--r-- 0 hdfs hdfs 0 mapreduce.tar.gz

Resolution

Restore MapReduce Archive

Copy the archive from a healthy local installation:

xxxxxxxxxx

hdfs dfs -put -f \

/usr/odp/<version>/hadoop/mapreduce.tar.gz \

/odp/apps/<version>/mapreduce/

Restore Tez Archive

xxxxxxxxxx

hdfs dfs -put -f \

/usr/odp/<version>/tez/tez.tar.gz \

/odp/apps/<version>/tez/

Verify:

xxxxxxxxxx

hdfs dfs -ls /odp/apps/<version>/tez

hdfs dfs -ls /odp/apps/<version>/mapreduce

Confirm the files are no longer zero bytes.

Validation

Test Tez

Run the Tez service check from Ambari.

Test MapReduce

xxxxxxxxxx

yarn jar \

/usr/odp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar \

pi 2 10

Both service checks should complete successfully.

Best Practices

• Verify /odp/apps/<version>/ contents immediately after upgrade.
• Validate archive file sizes before running service checks.
• Ensure HDFS permissions remain intact.
• Keep a backup of application archives prior to major upgrades.