Troubleshooting the Kerberos Configuration Issues in Ambari

This documentation provides detailed guidance to diagnose and resolve the Kerberos configuration issues in Ambari. It is intended to support both technical teams and system administrators in efficiently troubleshooting failures when disabling Kerberos or when the service is missing or partially uninstalled.

This troubleshooting page aims to provide a structured approach to:

• Quickly identify and diagnose common Kerberos issues in Ambari.
• Systematically resolve problems by using detailed log analysis, DNS validation, connectivity checks, and service restoration commands.
• Streamline the support process by including clear steps and commands to address the Kerberos-related failures.
• Minimize downtime by enabling technical teams to act on issues without needing extensive external support.

Issue Overview

1. Kerberos Disable Failure: Service startup might be affected if the process to disable Kerberos in Ambari is not completed successfully.
2. Kerberos Service Missing: In some cases, even if the disable operation appears partially successful, the Kerberos service might be removed from the cluster configuration.

The following steps outline the procedures to identify the root cause, correct configuration issues, and restore the Kerberos setup.

Verify the Kerberos Disable Failure

1. Check the Ambari UI and Server Logs:

   1. Ambari UI: Navigate to the Ambari Operations (OPS) section and look for the failed operation that indicates the Kerberos disable failure.
   2. Ambari Server Logs: Check the Ambari server logs (/var/log/ambari-server/ambari-server.log) to identify the cause of the failure. Look for errors related to Kerberos operations or authentication issues.
   3. Possible Cause: If the issue is related to authentication failure (e.g., Authfailed in the Zookeeper zone), it may be necessary to temporarily add the skipacl property to bypass the ACL checks.

2. Fix Kerberos Disable Failure:

   1. If the Kerberos disable fails due to an authentication issue, perform the following steps to add the skipacl property. Once Kerberos is successfully disabled, you can remove the skipacl setting.

3. Check the DNS Configuration: Ensure that all nodes in the cluster have correct DNS settings, as DNS issues can significantly affect the Kerberos authentication.

   1. Forward and Reverse DNS Lookups:

      • Verify the hostname to IP address resolution: nslookup <hostname>
      • Verify the IP address to hostname resolution: nslookup <IP-address>

   2. Hostname Consistency:

      • Ensure that the reverse DNS lookup result matches the fully qualified domain name (FQDN) used by the Kerberos service.

   3. Update /etc/hosts (if necessary): If DNS inconsistencies are found, update the /etc/hosts file on each node to include accurate IP-to-hostname mappings to ensure smooth communication.

4. Configure the Database Adjustments:

   1. Set Security Type to Kerberos

      1. Log in to the Ambari database and check the current security type: SELECT security_type FROM clusters;.
      2. Update the security type to Kerberos: UPDATE clusters SET security_type='KERBEROS';.

   2. Update the kerberos-env Configuration:

      1. Check the current configuration for kerberos-env: SELECT version_tag, type_name, selected, create_timestamp FROM clusterconfig WHERE type_name = 'kerberos-env';.
      2. Update to the latest version tag for kerberos-env:

SET @latest_version_tag := (

    SELECT version_tag

    FROM clusterconfig

    WHERE type_name = 'kerberos-env'

    ORDER BY create_timestamp DESC

    LIMIT 1

);

​

UPDATE clusterconfig

SET selected = 1

WHERE type_name = 'kerberos-env' AND version_tag = @latest_version_tag;

3. Update the krb5-conf Configuration:
   1. Check the current configuration for krb5-conf: SELECTversion_tag, type_name, selected, create_timestampFROMclusterconfigWHEREtype_name = 'krb5-conf';.
   2. Update to the latest version tag for krb5-conf:

SET @latest_version_tag := (

    SELECT version_tag

    FROM clusterconfig

    WHERE type_name = 'krb5-conf'

    ORDER BY create_timestamp DESC

    LIMIT 1

);

​

UPDATE clusterconfig

SET selected = 1

WHERE type_name = 'krb5-conf' AND version_tag = @latest_version_tag;

Restart the Ambari Server

After updating the configurations, restart the Ambari Server to apply the changes.

ambari-server restart

Adding the Kerberos Service using the Curl Commands

1. Retrieve the Cluster Name Automatically: Use the following command to dynamically retrieve the cluster name and store it in a variable to be used in the subsequent steps:

# Retrieve the cluster name dynamically

CLUSTER_NAME=$(curl -s -u admin:admin -H 'X-Requested-By: ambari' -X GET http://$(hostname -f):8080/api/v1/clusters | grep -oP '(?<="cluster_name" : ")[^"]*')

​

# Check if the cluster name was retrieved successfully

if [ -z "$CLUSTER_NAME" ]; then

  echo "Error: Could not retrieve the cluster name. Please check Ambari connectivity."

  exit 1

else

  echo "Cluster Name: $CLUSTER_NAME"

fi

2. Add the Kerberos Service and Components: Execute the following curl commands to add the Kerberos service back to your cluster using the automatically retrieved cluster name.

# Add Kerberos service to the cluster

curl -u admin:admin -i -H "X-Requested-By: ambari" -X POST -d '{"ServiceInfo":{"service_name":"KERBEROS"}}' http://$(hostname -f):8080/api/v1/clusters/$CLUSTER_NAME/services

​

# Add Kerberos client component to the service

curl -u admin:admin -H "X-Requested-By: ambari" -X POST http://$(hostname -f):8080/api/v1/clusters/$CLUSTER_NAME/services/KERBEROS/components/KERBEROS_CLIENT

​

# Get the list of hosts and save to a file

curl -s -u admin:admin http://$(hostname -f):8080/api/v1/hosts | grep host_name | sed -n 's/.*"host_name" : "\([^\"]*\)".*/\1/p' > hostcluster.txt

​

# Add Kerberos client component to each host in the cluster

for i in $(cat hostcluster.txt); do 

    curl -u admin:admin -H "X-Requested-By: ambari" -X POST http://$(hostname -f):8080/api/v1/clusters/$CLUSTER_NAME/hosts/$i/host_components/KERBEROS_CLIENT; 

done

​

Ex:

curl -H "X-Requested-By:ambari" -u admin:admin -X POST -d '{ "Credential" : { "principal" : "admin/admin@DOMAIN.COM", "key" : "PASSW0RD", "type" : "temporary" } }' http://$(hostname -f):8080/api/v1/clusters/$CLUSTER_NAME/credentials/kdc.admin.credential

3. Setting Up the Temporary KDC Credentials: To proceed with further Kerberos operations, follow these steps to set up temporary KDC credentials:

   1. Access the Ambari UI:

      • Navigate to HOSTS.
      • Go to the Kerberos Client section.
      • Click on Install.

   2. Store Kerberos Credentials:

      • This process prompts you for Kerberos credentials.
      • Once the credentials are successfully stored in Ambari, proceed with the following command.

# Retrieve the cluster name dynamically

CLUSTER_NAME=$(curl -s -u admin:admin -H 'X-Requested-By: ambari' -X GET http://$(hostname -f):8080/api/v1/clusters | grep -oP '(?<="cluster_name" : ")[^"]*')

​

# Check if the cluster name was retrieved successfully

if [ -z "$CLUSTER_NAME" ]; then

  echo "Error: Could not retrieve the cluster name. Please check Ambari connectivity."

  exit 1

else

  echo "Cluster Name: $CLUSTER_NAME"

fi

Final Step: Adding the Clients back to the Ambari UI

After setting up the temporary KDC credentials, execute the following commands to add the clients back to the Ambari UI:

# Set the host components state to INSTALLED

curl -u admin:admin -H 'X-Requested-By: ambari' -X PUT -d '{"HostRoles": {"state":"INSTALLED"}}' http://$(hostname -f):8080/api/v1/clusters/$CLUSTER_NAME/host_components?HostRoles/state=INIT

​

​

# optional

# Set the cluster's security type back to Kerberos and regenerate keytabs

curl -u admin:admin -H "X-Requested-By: ambari" -X PUT \

-d '{"Clusters": {"security_type": "KERBEROS"}}' \

http://$(hostname -f):8080/api/v1/clusters/$CLUSTER_NAME?regenerate_keytabs=all

Conclusion

This documentation provides a step-by-step approach to resolve the Kerberos configuration issues in Ambari. It includes automated steps to fetch the cluster name, diagnose the failure to disable Kerberos, correct the database settings, restore the Kerberos service, and ensure that all necessary components are added back to the cluster.

If these steps do not fully resolve the issue, further investigation into Ambari server logs and network settings might be necessary.

Additional Troubleshooting Steps

1. Do the detailed Log Analysis

   1. Check the Ambari Server Logs

      1. The location of logs: /var/log/ambari-server/ambari-server.log
      2. Look for specific error messages related to the Kerberos setup or service failures.
      3. Common errors to search for:
         • Authfailed
         • Kerberos authentication failed
         • Unable to communicate with KDC

   2. Check the Kerberos Client Logs

      1. The Kerberos client logs might contain valuable information if the client fails to communicate with the KDC.
      2. The location of logs: /var/log/krb5kdc.log and /var/log/kadmin.log on the KDC server.
      3. Search for errors related to authentication, ticket expiration, or encryption.

2. Validate the Kerberos Ticket Granting

   1. Check Kerberos Ticket Availability

      1. Ensure that a valid Kerberos ticket is available for the current session.
      2. Use the this command to check if a ticket is present: klist
      3. If no ticket is listed, obtain a new Kerberos ticket using: kinit admin/admin@YOUR_REALM
      4. If ticket acquisition fails, it may indicate an issue with the KDC or the credentials provided.

   2. Check Keytab File Permissions

      1. Verify that the keytab files have the correct permissions and are accessible to the Ambari services.
      2. The common keytab file locations: /etc/security/keytabs/.
      3. Ensure that the permissions allow the Ambari user to access these files: ls -l /etc/security/keytabs/.

3. Verify the Network Connectivity

   1. Verify the connectivity to KDC

      1. Ensure that all nodes can communicate with the Kerberos Key Distribution Center (KDC).
      2. Use the this command to check connectivity: nc -zv <KDC-Hostname> 88. The port 88 is the default port for Kerberos communication.
      3. If connectivity fails, check the firewall settings or network restrictions between nodes.

   2. Validate the DNS Configuration

      1. Ensure that all nodes in the cluster have proper DNS resolution, as Kerberos heavily relies on DNS.
      2. Forward and Reverse the DNS Lookups.

nslookup <hostname>

nslookup <IP-address>

4. Verify the Ambari Configuration

   1. Confirm the Kerberos Properties in Ambari:

      1. Validate that the Kerberos-related properties are correctly configured in Ambari.
      2. Navigate to Ambari Web UI > Admin > Kerberos and check the following properties:
         • KDC Host
         • Realm Name
         • Kadmin Host
         • Encryption Types

   2. Check for the Missing or Incorrect Kerberos Components:

      1. Ensure that all necessary components are installed:
         1. Kerberos client packages (krb5-workstation, krb5-libs)
         2. Proper entries in /etc/krb5.conf

   3. Synchronize the Time across Nodes:

      1. The Time synchronization is critical for the Kerberos authentication.
      2. Use ntp or chrony to ensure that all nodes in the cluster have synchronized clocks: ntpq -p.
      3. The difference in time between nodes can cause the Kerberos ticket validation errors.

5. Common Kerberos Configuration Issues for MIT KDC

   1. Check for Duplicate Service Principal Names (SPNs):

      1. Duplicate SPNs can cause authentication failures.
      2. List all principals in the KDC database: kadmin.local -q "listprincs".
      3. Ensure that there are no duplicate entries for the same service.

   2. Validate the Encryption Types:

      1. Verify that the encryption types used by Kerberos are compatible with your cluster's security settings.
      2. Check the supported encryption types in /etc/krb5.conf:

[libdefaults]

    default_tkt_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96

    default_tgs_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96

6. Re-initializing the Kerberos Configuration in Ambari
   1. Reset and reconfigure kerberos in Ambari: If other troubleshooting steps fail, it might be necessary to completely re-initialize the Kerberos configuration in Ambari:
      1. Disable Kerberos in Ambari from the Admin > Kerberos section.
      2. Remove Kerberos-related configurations from /etc/krb5.conf and /etc/security/keytabs/.
      3. Re-enable Kerberos in Ambari using the setup wizard to ensure that all necessary configurations are applied correctly.