Error in Submitting YARN Job

The logs indicate that there is an issue with the user jupyterhub being used to submit applications on YARN. Specifically, the error message highlights:

Requested user jupyterhub is not whitelisted and has id 118, which is below the minimum allowed 1000.

This suggests that YARN has a policy restricting users with IDs below 1000, which is common on many Linux systems, where system users (with IDs below 1000) are restricted from running applications on YARN.

To resolve this issue, you have a couple of options:

1. Change the jupyterhub user ID:
   • If the jupyterhub user has an ID below 1000, you can change the user ID of jupyterhub to a value greater than 1000:

sudo usermod -u 1001 jupyterhub

- After changing the user ID, you will need to update any file permissions or group memberships associated with this user.

2. Check User Whitelisting for YARN:
   • In some environments, YARN has strict user whitelisting policies. You may need to explicitly add jupyterhub to the list of allowed users in YARN's security settings, which might involve modifying yarn-site.xml or contacting your system administrator to allow the user.

Additional Information:

• After making changes to YARN configurations, make sure to restart the YARN ResourceManager and NodeManagers for the changes to take effect.
• If you're using a Kerberos-secured environment (as indicated by your log), ensure that the user jupyterhub has the correct Kerberos principal and permissions to submit jobs.

By addressing these configurations, you should be able to submit applications as the jupyterhub user without encountering this error.