Securing NiFi with Existing CA Certificates

This document explains how to secure NiFi with existing Certificate Authorities (CA) certificates (without NiFi LDAP Authentication).

In this configuration, commercial Certificate Authorities (CAs) play a pivotal role. Alternatively, an enterprise CA with an internal CA certificate trusted by corporate devices can be utilized. Several commercial vendors offer signed certificates, including TinyCert, chosen here for its cost-effectiveness. Another option, Let’s Encrypt, provides free certificates but requires hostname ownership verification, which involves additional steps not covered in this scenario.

All certificates and keys in this setup adhere to the Privacy-Enhanced Mail (PEM) format, widely recognized as a standard for certificates and keys. PEM utilizes Base64 encoding for file contents, ensuring seamless transmission across diverse mediums and storage systems. These files can be easily inspected using plain text editors, and tools such as openssl and keytool are proficient in parsing them.

Example PEM file:

Bash
Copy

Example parsed contents:

Bash
Copy

The prerequisites for this scenario (issued by the IT department):

  • A signed NiFi server certificate for the designated host (e.g., secure.nifi) in PEM format (nifi.pem)
  • The corresponding private key in PEM format (nifi.key)
  • A signed client certificate for the designated user (e.g., CN=my_username, …​) in PEM format (client.pem)
  • The corresponding private key in PEM format (client.key)
  • The CA certificate in PEM format (cacert.pem)

For more information on converting certificates between various forms, see Toolkit Guide: Additional Certificate Commands.

The final result will include a self-signed external CA (the root), a keystore and truststore with the required certificates for the NiFi instance to function, and a client certificate for a user to access NiFi.

Certificate Requirements

Ensure compliance with the following certificate requirements, maintaining the sequence but shuffling the order:

  • The certificate's signature algorithm must utilize sha256WithRSAEncryption (SHA-256).

  • Subject Alternate Names (SANs) are obligatory and must minimally encompass the Fully Qualified Domain Name (FQDN) of the host.

  • Include the FQDN utilized for the Common Name (CN) as a DNS SAN entry.

  • Supplemental names for the certificate/host can be appended as SANs.

  • Avoid the usage of wildcard certificates. Each cluster node should possess its own certificate. In cases where NiFi or NiFi Registry operates behind Knox, refrain from deploying wildcard certificates for Knox.

  • If contemplating the utilization of a load balancer for the NiFi service, incorporate the FQDN for the load balancer as a DNS SAN entry.

  • The X509v3 KeyUsage segment of the certificate should incorporate the subsequent attributes:

    • DigitalSignature
    • Key_Encipherment

  • The X509v3 ExtendedKeyUsages segment of the certificate necessitates the inclusion of the ensuing attributes:

    • clientAuth: Intended for TLS web client authentication.
    • serverAuth: Intended for TLS web server authentication.

  • The KeyStore should exclusively hold a single PrivateKeyEntry. The utilization of multiple private keys within one KeyStore is not endorsed.

  • The KeyStore password and the key/certificate password must either match or remain unset.

  • Consistency in KeyStore password and key/certificate password is important across all distinct KeyStores utilized on each NiFi cluster node. Individual passwords for NiFi hosts are not supported by Ambari.

Generate keystore.jks and truststore.jks for NiFi Hosts

To generate keystore.jks and truststore.jks for NiFi hosts, perform the following:

  1. Determine if the server certificate (nifi.pem) contains the complete certificate chain or just the server certificate. If the sequence -----BEGIN CERTIFICATE-----occurs only once, this is just the server certificate. If it occurs multiple times, the certificate chain is present. If the certificate chain is present, continue from Step 3 else continue to Step 2.
Bash
Copy

The above code prints the Base64-encoded contents.

  1. Concatenate the server certificate and CA certificate to form the certificate chain.
Bash
Copy

The above code concatenates the CA certificate to the NiFi server certificate in the proper order.

  1. Form the PKCS12 keystore from the certificate chain and private key.
  • The following code generates a PKCS12 keystore that includes the private key and certificate chain under the alias nifi-key. This command will prompt you to enter an export password. Choose a secure password and confirm it by entering it twice (for this example, an insecure password 'passwordpassword' is used).
Bash
Copy
  • Verifiy the contents of the PKCS12 keystore (optional). The following command will prompt for the same password entered in the previous step.
Bash
Copy
  1. Convert the PKCS12 keystore for the NiFi instance into the Java KeyStore file (keystore.jks). PKCS12 keystores are usable by NiFi, but JKS format is handled more robustly and causes fewer edge cases. JKS keystores cannot be formed directly from PEM files, so the PKCS12 keystore serves as an intermediate form.
Bash
Copy

This command converts the PKCS12 keystore to a JKS keystore. It will prompt you to enter a new keystore password twice and then prompt you to enter the password set on the PKCS12 keystore in the previous step.

  1. Convert the CA certificate into the NiFi truststore (truststore.jks) to allow trusted incoming connections.
Bash
Copy

This command imports the CA certificate into the truststore and prompts for a new truststore password twice.

  1. (Optional) Move the keystore.jks and truststore.jks files into the conf/ directory. This step is not required, but helps with organization.
  2. Generate client certificates, and add it to machine browsers to access NiFi UI. Generate the client certificate keystore from the client certificate and key.
Bash
Copy

This command creates the PKCS12 keystore containing the client certificate and private key. This command prompts for an export password twice.

From here on, the instructions are the same as those for the TLS Toolkit, starting with loading the client certificate in the browser.

Enable SSL with Existing Certificates (Using Ranger for Authorization)

To enable SSL with existing certificates, perform the following:

  1. Select the "Enable SSL?" checkbox. Navigate to Ambari UI > NiFi > Configs > Advanced > Advanced nifi-ambari-ssl-config > Enable SSL? checkbox.

  2. Enter the values for the Keystore path, Keystore password, and Keystore type.

    • Keystore path: /etc/nifi/conf/keystore.jks

  3. Enter the values for the Truststore path, Truststore password, and Truststore type.

    • Truststore path: /etc/nifi/conf/truststore.jks
Advanced nifi-ambari-ssl-config

Advanced nifi-ambari-ssl-config

Setting Up Identity Mapping

About This Task

You can use identity mapping properties to normalize user identities. Once set up, NiFi treats identities authenticated by different identity providers (certificates, LDAP, Kerberos) the same. This prevents the creation of duplicate users and ensures user-specific configurations, such as authorizations, need to be set up only once per user.

Perform the following steps:

  1. From the NiFi service Configs tab, click Advanced nifi-properties.
  2. Use the Filter box to search for nifi.security.identity.mapping.pattern.
  3. Enter the following values:
FieldSample Value
nifi.security.identity.mapping.pattern.dn^CN=(.*?), OU=(.*?)$
nifi.security.identity.mapping.value.dn$1@$2
nifi.security.identity.mapping.pattern.kerb^(.*?)/instance@(.*?)$
nifi.security.identity.mapping.value.kerb$1@$2
  1. Click Save.
  2. Restart NiFi using the Restart all Required option from the Action menu.
Advanced nifi-properties

Advanced nifi-properties

Logging into NiFi After Enabling SSL

Once SSL is enabled in NiFi, perform the following steps to securely log in:

  1. Open NiFi from the Ambari Quick Links menu.
  2. When prompted by the browser, select the certificate you recently imported. Use incognito mode if the certificate prompt does not appear.

When NiFi runs on a host with Ambari and SSL enabled, the default URL becomes secure: https://<local-host>:9091/nifi.

Additional Steps

Ranger Test Connection Failure Due to SSL

Issue:

"Connection Failed. Unable to retrieve any resources using given parameters. java.net.SocketException: java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: sun.security.ssl.SSLContextImpl$DefaultSSLContext)"

Solution:

Perform the following steps to review the configurations:

  1. Login into Ranger UI > Edit Nifi Service Repo > Review “NiFi URL”
  2. Review “Config Properties” added/updated correctly.

Ranger Test Connection Failure Due to permission 403 forbidden

Issue:

NiFi authentication fails due to insufficient permissions, resulting in an access denied error in the nifi-user.log.

Log sample:

plaintext
Copy

Solution:

  1. Review the NiFi Log:
Bash
Copy

Rerun the Ranger NiFi Test connection and observe the log for authentication events.

  1. Add User in Ranger UI:

    1. Log in to the Ranger UI.

    2. Navigate to Settings > Users.

    3. Click on Add New User.

    4. Enter the following details:

      • User Name: Full Certificate DN (e.g., CN=ambari007-ha.adsre.com, O=ACCELDATA, L=PA, ST=LA, C=US)
      • Password: Set a dummy password that meets the password requirements.

    5. Click Save.

  2. Assign NiFi Policy:

    1. Go to Ranger NiFi Policy.

    2. Add the new user to an existing policy or create a new one.

      • Example New Policy:
        • NiFi Resource Identifier: /*
        • Select User: The newly added user.

    3. Save the policy.

These steps will ensure that the specified user has the required permissions to access the requested resources in NiFi.

Insufficient Permissions | No Applicable Policy Could Be Found

Issue:

"Insufficient Permissions | No applicable policies could be found. Contact the system administrator."

Solution:

Review the Ranger Audit Configs by performing the following:

  1. Add User in Ranger: User: CN=nifi_user, O=ACCELDATA, L=PA, ST=LA, C=US

    • Log in to the Ranger UI.

    • Navigate to Settings > Users.

    • Click on Add New User.

    • Enter the following details:

      • User Name: CN=nifi_user, O=ACCELDATA, L=PA, ST=LA, C=US
      • Password: Set a dummy password that meets the password requirements.

    • Click Save.

  2. Assign NiFi Policy:

    • Navigate to Ranger NiFi Policy.
    • Add the new user to an existing policy or create a new one.
      • Example New Policy:
        • NiFi Resource Identifier: /flow
Audit Access Log Detail

Audit Access Log Detail

Enabling SSL with Existing Certificates (Without Using Ranger for Authorization)

  1. Enable SSL:

    • Tick the "Enable SSL?" checkbox.

  2. Configure Keystore:

    • Specify the following values:
      • Keystore path: /etc/nifi/conf/keystore.jks
      • Keystore password
      • Keystore type

  3. Configure Truststore:

    • Set the following values:
      • Truststore path: /etc/nifi/conf/truststore.jks
      • Truststore password
      • Truststore type

  4. Provide Initial Admin Identity:

    • The Initial Admin Identity is required for the initial administrator to access the UI and create additional users, groups, and policies. This step is necessary unless using the Ranger plugin for NiFi authorization.
    • Format: CN=admin, OU=NIFI
    • After setting the Initial Admin Identity, generate a certificate for this user.

  5. Define Node Identities:

    • This step specifies the identity of each node in a NiFi cluster, enabling communication between clustered nodes. It is mandatory unless using the Ranger plugin for NiFi authorization.

    • Example properties:

      • <property name="Node Identity 1">CN=node1.fqdn, OU=NIFI</property>
      • <property name="Node Identity 2">CN=node2.fqdn, OU=NIFI</property>
      • <property name="Node Identity 3">CN=node3.fqdn, OU=NIFI</property>

    • Replace node1.fqdn, node2.fqdn, and node3.fqdn with their respective fully qualified domain names (FQDN).

Type to search, ESC to discard
Type to search, ESC to discard
Type to search, ESC to discard
  Last updated on Oct 1, 2024
Next to read:
Configuring SSL for the Ambari Server
On This Page
Securing NiFi with Existing CA CertificatesCertificate RequirementsGenerate keystore.jks and truststore.jks for NiFi HostsEnable SSL with Existing Certificates (Using Ranger for Authorization)Setting Up Identity MappingLogging into NiFi After Enabling SSLAdditional StepsRanger Test Connection Failure Due to SSLRanger Test Connection Failure Due to permission 403 forbiddenInsufficient Permissions | No Applicable Policy Could Be FoundEnabling SSL with Existing Certificates (Without Using Ranger for Authorization)