Services SSL and Truststore Configuration

Configuring the Truststore for Each Service

The truststore holds the CA certificates that each Hadoop service needs to trust other services or external clients.

1. Import the CA Certificate: Use the keytool command to import the CA certificate into each service’s truststore:

keytool -import -alias cacert -file /path/to/ca.pem -keystore /path/to/service-truststore.jks

Example for NameNode:

keytool -import -alias cacert -file /path/to/ca.pem -keystore /path/to/namenode-truststore.jks

2. Common Issues and Fixes:

   1. Invalid Truststore Path: Ensure the truststore path is correct. If the truststore file doesn't exist, keytool will create it.
   2. Truststore Password Errors: Ensure that the truststore password is correct and set in the relevant service configuration file (e.g., core-site.xml). If the password is incorrect, the service will fail to load the truststore.

3. Verify CA Certificate in Truststore: After importing the certificate, you can list the certificates in the truststore to verify.

4. The output must display the alias (e.g., cacert) and details of the imported certificate.

Update the Hadoop Service Configuration Files

Each Hadoop service (e.g., NameNode, DataNode, and ResourceManager) requires its configuration files to be updated to enable SSL and specify the correct keystore and truststore paths.

1. Edit the Service Configuration (e.g., core-site.xml ): Add or update the following properties in each service’s configuration file:

<property>

   <name>hadoop.ssl.enabled</name>

   <value>true</value>

</property>

<property>

   <name>hadoop.ssl.keystore</name>

   <value>/path/to/service-keystore.jks</value>

</property>

<property>

   <name>hadoop.ssl.keystore.password</name>

   <value>your-keystore-password</value>

</property>

<property>

   <name>hadoop.ssl.truststore</name>

   <value>/path/to/service-truststore.jks</value>

</property>

<property>

   <name>hadoop.ssl.truststore.password</name>

   <value>your-truststore-password</value>

</property>

2. Key Points to Check:

   1. Ensure the correct path to the keystore and truststore files.
   2. Double-check the keystore and truststore passwords. If passwords are incorrect or missing, SSL initialization fails.

3. Common Issues and Fixes:

   1. Misconfigured File Paths: Ensure the paths to the keystore and truststore are correct in the service configuration. If the service can’t find the keystore/truststore, it fails to start.
   2. Password Errors: Make sure the correct passwords for the keystore and truststore are specified. If the password is wrong, the Hadoop services fail to load SSL certificates.

Restart the Hadoop Services

After updating the configuration files, you must restart each Hadoop service to apply the changes.

Restart the Services: Use Ambari or the Hadoop command line to restart each service (e.g., NameNode, DataNode, and ResourceManager):

1. You can also use Ambari to stop and start services via the web UI.
2. Check the Service Status: After restarting, verify that each service started correctly by checking the service logs. The logs can be found in /var/log/hadoop/<service-name>.

Testing the SSL Configuration

Once all the services have been restarted with SSL enabled, it’s essential to test SSL connections to verify the setup.

1. Test the SSL Connection to Hadoop Services: Use curl or openssl to verify that the services are accessible over SSL. For example, to test the NameNode UI:

curl -v -k https://<namenode-host>:9871

2. If the SSL setup is correct, you should see the certificate and be able to connect securely. If the connection fails, look for SSL-related errors (e.g., untrusted certificate, handshake failure, etc.).
3. Check Logs for SSL Errors: If services fail to start or connections fail, check the service logs (e.g., namenode.log, datanode.log, etc.) for SSL errors. Common errors include:
   1. Certificate Mismatch: If the service certificate’s Common Name (CN) does not match the hostname, SSL connections may fail.
   2. Truststore Issues: If the truststore is misconfigured or missing the CA certificates, services won’t be able to validate certificates from other services or clients.

Common SSL Configuration Issues and Fixes

1. Certificate Expiration:

   • Check if any service or CA certificates have expired. Use keytool -list -v to inspect the expiration dates of certificates in the keystore or truststore.
   • Renew any expired certificates and update the keystore/truststore accordingly.

2. Hostname Mismatch:

   • Ensure the Common Name (CN) in the service certificate matches the service’s fully qualified domain name (FQDN).
   • If the CN is incorrect, the SSL handshake failures occur.

3. Truststore Password Errors:

   • Ensure the truststore password in the service configuration is correct. If the wrong password is set, the service won’t be able to load the truststore, leading to SSL failures.

4. Invalid Truststore or Keystore Paths:

   • Double-check that the file paths to the keystore and truststore in the configuration files (e.g., core-site.xml, ssl-server.xml) are accurate and accessible.