Configure Pulse on Kubernetes to monitor a standalone NiFi cluster

Configure Pulse on Kubernetes to monitor a standalone Apache NiFi cluster.

This is supported from Pulse 4.1.2 onwards.

Prerequisites

Before you begin, ensure that:

Pulse Manager is accessible and you have administrator credentials for API access.
Pulse is installed and configured on Kubernetes.
The Pulse database node is configured and accessible.
Container registry credentials are configured as a Kubernetes secret.
The elastic-certificates.p12 certificate file is available.
The standalone NiFi cluster and NiFi Registry are accessible from Pulse Manager.
Host mappings are configured for the Pulse and NiFi nodes.

Replace the sample credentials, URLs, hostnames, IP addresses, and file paths in this procedure with values from your environment. Do not store unencrypted credentials in source control.

Create the NiFi Cluster Configuration

Create a file named nifi_inputs.json with the following configuration:

{ "standAloneDistroType": "Nifi", "standAloneComponents": "Nifi", "isEdgeNode": false, "clusterType": "Stand-Alone", "clusterName": "ad-nifi", "clusterDisplayName": "ad-nifi", "clusterOriginalName": "ad-nifi", "isKerberosEnabled": false, "isParcelBasedDeployment": false, "SSHKeyAlgo": "RSA", "httpsEnabledInCluster": false, "sshUser": "root", "zkJMXPort": "9999", "enableLogSearch": true, "zkEnabled": false, "nifiZKEnabled": true, "nifiRegistryInstalled": true, "nifiRegistryURL": "http://odp901.acceldata.dvl:18080,http://odp902.acceldata.dvl:18080,http://odp903.acceldata.dvl:18080" }

If NiFi uses embedded ZooKeeper, set nifiZKEnabled to false and skip the ZooKeeper host upload step.

Create the NiFi Hosts File

Create a file named nifihosts.yml to specify the NiFi endpoints and credentials:

hosts: - https://odp901.acceldata.dvl:8443 - https://odp902.acceldata.dvl:8443 - https://odp903.acceldata.dvl:8443 auth: username: admin password: <ENCRYPTED_PASSWORD>

Encrypt the NiFi Password

Use Pulse Manager to encrypt the NiFi user password:

curl -X POST \ -u "pulse:<PULSE_API_PASSWORD>" \ -H 'Content-Type: application/json' \ --data '{"encrypt":"<NIFI_PASSWORD>"}' \ http://odp702.acceldata.dvl:30730/manager/api/v1/mserver/admin/encrypt

Copy the encrypted password returned by the command and specify it as the auth.password value in nifihosts.yml.

Upload the Host Configuration

If NiFi uses an external ZooKeeper, upload the ZooKeeper hosts file:

curl -X PUT \ -u "pulse:<PULSE_API_PASSWORD>" \ -F "zkhosts=@/root/nifizookeeper" \ http://odp702.acceldata.dvl:30730/manager/api/v1/mserver/upload/zkhosts/ad-nifi

Upload the NiFi hosts file:

curl -X PUT \ -u "pulse:<PULSE_API_PASSWORD>" \ -F "nifihosts=@/root/nifihosts.yml" \ http://odp702.acceldata.dvl:30730/manager/api/v1/mserver/upload/nifihosts/ad-nifi

Replace:

ad-nifi with your cluster name.
/root/nifizookeeper with the path to your ZooKeeper hosts file.
/root/nifihosts.yml with the path to your NiFi hosts file.

Initialize Pulse Manager

Initialize Pulse Manager with the required version, container registry, and retention settings:

curl --request POST \ --url http://odp702.acceldata.dvl:30730/manager/api/v1/mserver/init \ -u "pulse:<PULSE_API_PASSWORD>" \ --header 'Content-Type: application/json' \ --data '{ "pulse_version": "testBuild", "registry_url": "191579300362.dkr.ecr.us-east-1.amazonaws.com/acceldata", "hdfs_retention_days": 31, "mongo_retention_days": 32, "logstore_retention": 32, "metrics_retention": 32, "registry_secret": "adreg-docker-secret" }'

Ensure that the adreg-docker-secret Kubernetes secret exists in the Pulse namespace and contains valid container registry credentials.

Upload the Elastic Certificate

Upload the elastic-certificates.p12 certificate for secure Elastic connectivity:

curl --request PUT \ --url http://odp702.acceldata.dvl:30730/manager/api/v1/mserver/upload/certs/es \ -u "pulse:<PULSE_API_PASSWORD>" \ --header 'Content-Type: multipart/form-data' \ --form es-certs.p12=@elastic-certificates.p12

Configure the NiFi Cluster

Configure the NiFi cluster using the nifi_inputs.json file:

curl --request POST \ --url http://odp702.acceldata.dvl:30730/manager/api/v1/mserver/config/cluster \ -u "pulse:<PULSE_API_PASSWORD>" \ --header 'Content-Type: application/json' \ --data @nifi_inputs.json

Deploy the Database Components

Deploy the Pulse database components:

curl --request PUT \ --url http://odp702.acceldata.dvl:30730/manager/api/v1/mserver/deploy/db \ -u "pulse:<PULSE_API_PASSWORD>"

Push the database configuration to the NiFi cluster:

curl --request PUT \ --url http://odp702.acceldata.dvl:30730/manager/api/v1/mserver/dbpush/ad-nifi \ -u "pulse:<PULSE_API_PASSWORD>" \ --header 'Content-Type: application/json' \ --data '{}'

Replace ad-nifi with your cluster name.

Deploy Pulse Core Services

Deploy the Pulse core services and specify the required host mappings:

curl --request PUT \ --url http://odp702.acceldata.dvl:30730/manager/api/v1/mserver/deploy/core \ -u "pulse:<PULSE_API_PASSWORD>" \ --header 'Content-Type: application/json' \ --data '{ "extra_hosts": { "10.100.10.99": "odppulse901.acceldata.dvl", "10.100.10.169": "odp901.acceldata.dvl", "10.100.10.170": "odp902.acceldata.dvl", "10.100.10.172": "odp903.acceldata.dvl" } }'

Replace the sample IP addresses and FQDNs with the values for your environment.

Important: Ensure that the IP address-to-FQDN mappings are correct. Incorrect mappings can cause connectivity failures or TLS hostname mismatches.

Configure Log Collection

Configure the Pulse log collector to collect NiFi and system logs.

In the log collector configuration, such as vars.yml, add the required log locations:

log_locations: kern: - path: /var/log/kern.log type: DATESTAMP nifi: - path: /root/nifi-setup/nifi-2.7.2.3.3.6.4-1/logs/nifi-app.log,/root/nifi-setup/nifi-2.7.2.3.3.6.4-1/logs/nifi-user.log,/root/nifi-setup/nifi-2.7.2.3.3.6.4-1/logs/nifi-bootstrap.log,/root/nifi-setup/nifi-registry-2.7.2.3.3.6.4-1/logs/nifi-registry-app.log,/root/nifi-setup/nifi-registry-2.7.2.3.3.6.4-1/logs/nifi-registry-bootstrap.log,/root/nifi-setup/nifi-registry-2.7.2.3.3.6.4-1/logs/nifi-registry-event.log type: DEFAULT syslog: - path: /var/log/syslog,/var/log/messages type: DATESTAMP zookeeper: - path: /zookeeper.log type: DEFAULT

Update the NiFi version directories and log paths to match your installation.

Reconfigure the Cluster

Apply the updated configuration to the cluster:

curl --request POST \ --url http://odp702.acceldata.dvl:30730/manager/api/v1/mserver/reconfig/cluster/ad-nifi \ -u "pulse:<PULSE_API_PASSWORD>" \ --header 'Content-Type: application/json' \ --data '{ "forcePull": true }'

Replace ad-nifi with your cluster name.

Set forcePull to true when you want Kubernetes to pull the required container images again during reconfiguration.

Validate the Configuration

After you configure the cluster, verify the following:

Pulse Manager API requests complete successfully.
The database, core, and collector pods are running.
NiFi metrics and flows are available in the Pulse UI.
NiFi and system logs are available in Log Search.
NiFi Registry versioned flows are accessible.
NiFi Registry credentials are accepted.

Troubleshoot the Configuration

If the NiFi integration does not work as expected:

Verify the configuration file paths and cluster name.
Verify connectivity between Pulse Manager and the NiFi cluster.
Verify the NiFi credentials. If required, encrypt the password again and update nifihosts.yml.
Verify the host mappings, Kubernetes DNS configuration, and node connectivity.
Verify that the container registry secret exists and contains valid credentials.
Verify the log paths and file permissions.
Verify that the log collector pods can access the configured NiFi log locations.
If NiFi uses embedded ZooKeeper, ensure that nifiZKEnabled is set to false and skip the ZooKeeper host upload step.

Appendix: Command Reference

Use the following commands as a quick reference after you complete the configuration described in this guide.

1. Encrypt the NiFi Password

curl -X POST \ -u "pulse:<PULSE_API_PASSWORD>" \ -H 'Content-Type: application/json' \ --data '{"encrypt":"<PLAINTEXT_NIFI_PASSWORD>"}' \ http://odp702.acceldata.dvl:30730/manager/api/v1/mserver/admin/encrypt

Update nifihosts.yml with the encrypted password returned by the command.

2. Upload the ZooKeeper Hosts

If NiFi uses an external ZooKeeper, run:

curl -X PUT \ -u "pulse:<PULSE_API_PASSWORD>" \ -F "zkhosts=@/root/nifizookeeper" \ http://odp702.acceldata.dvl:30730/manager/api/v1/mserver/upload/zkhosts/ad-nifi

Skip this step if NiFi uses embedded ZooKeeper.

3. Upload the NiFi Registry Hosts

curl --request PUT \ --url http://odp702.acceldata.dvl:30730/manager/api/v1/mserver/upload/nifiregistryhosts/ad-nifi \ -u "pulse:<PULSE_API_PASSWORD>" \ --header 'Content-Type: multipart/form-data' \ --form nifiregistryhosts=@/root/nifihosts.yml

4. Initialize Pulse Manager

curl --request POST \ --url http://odp702.acceldata.dvl:30730/manager/api/v1/mserver/init \ -u "pulse:<PULSE_API_PASSWORD>" \ --header 'Content-Type: application/json' \ --data '{ "pulse_version": "testBuild-vips", "registry_url": "191579300362.dkr.ecr.us-east-1.amazonaws.com/acceldata", "hdfs_retention_days": 31, "mongo_retention_days": 32, "logstore_retention": 32, "metrics_retention": 32, "registry_secret": "adreg-docker-secret" }'

5. Upload the Elastic Certificate

6. Configure the NiFi Cluster

7. Deploy the Database Components

curl --request PUT \ --url http://odp702.acceldata.dvl:30730/manager/api/v1/mserver/deploy/db \ -u "pulse:<PULSE_API_PASSWORD>"

Push the database configuration:

curl --request PUT \ --url http://odp702.acceldata.dvl:30730/manager/api/v1/mserver/dbpush/ad-nifi \ -u "pulse:<PULSE_API_PASSWORD>" \ --header 'Content-Type: application/json' \ --data '{}'

8. Deploy Pulse Core Services

curl --request PUT \ --url http://odp702.acceldata.dvl:30730/manager/api/v1/mserver/deploy/core \ -u "pulse:<PULSE_API_PASSWORD>" \ --header 'Content-Type: application/json' \ --data '{ "extra_hosts": { "<IP Address 1>": "odppulse901.acceldata.dvl", "<IP Address 2>": "odp901.acceldata.dvl", "<IP Address 3>": "odp902.acceldata.dvl", "<IP Address 4>": "odp903.acceldata.dvl" } }'

9. Reconfigure the Cluster

After you update the required log paths, reconfigure the cluster:

Replace the sample cluster name, credentials, URLs, hostnames, IP addresses, registry information, and file paths with the values for your environment.

Last updated on

Was this page helpful?