Set Up NiFi in Pulse

This page explains how to set up NiFi in Pulse by installing the Acceldata NAR, configuring the reporting task, verifying data collection, and completing cluster setup in Pulse.

Info

The steps on this page apply to NiFi deployments on Ambari, Cloudera, Standalone, and other supported cluster types.

Configure NiFi Changes on NiFi Node and UI

Prerequisites

Obtain the ad-nifi-1.0.nar download link from the Acceldata team using the provided credentials.

Add NiFi NAR Files

On each NiFi node, copy the ad-nifi-1.0.nar file.
Place the file in the NiFi library classpath.

For example: /usr/odp/current/nifi/lib/

Verify the NiFi library location to confirm where the Acceldata NAR file must be placed.

Restart NiFi Services

Restart all NiFi services to load the NAR file.

Configure Acceldata Reporting Task

In the NiFi UI, navigate to Controller Settings > AccelDataReportingTask.

Set the property values as described below.

Field	Description
Monitor Group Name	Enter the cluster name as defined during Pulse installation.
Acceldata Influx Metric Collector URL	Add the VictoriaMetrics insert URL of the Acceldata NiFi Metric Collector.
Elasticsearch Concurrent Threads	Enter the number of threads to push data to Elasticsearch.
Elasticsearch Batch Size	Enter the Elasticsearch batch size per thread.
Max Provenance Events	Enter the number of events that trigger data collection.
Influx Batch Size	Enter the Influx batch size.
Influx Batch Flush Duration	Enter the flush duration for Influx batches, in milliseconds.

Verify Configuration

Check that VictoriaMetrics is updated with the prefix <groupname>.
Verify that Elasticsearch contains an index with the name <groupname>.

The prefix must correspond to the cluster name you added in Pulse, located under the Pulse node: $AcceloHome/work/<clustername>.

Configure NiFi in Pulse

Create a NiFi Hosts File

Info

This step applies only to NiFi on Standalone Clusters. > > Manually creating the NiFi hosts file applies only to standalone clusters. On ODP/HDP (Ambari) and CDP/CDH (CM), Pulse fetches host details automatically.

On the Pulse Node, create a nifihosts file in the directory /data01/acceldata/.
In the nifihosts file, add one NiFi IP or host per line with port details.

For example:

nifihosts1:9090 nifihosts2:9090

Note If the port is not specified, Pulse uses the default port (for example, 9090 for alerts endpoints).

Enter the /data01/acceldata/nifihosts path when asked during the Pulse configuration.

Configure Pulse Core Component

Run accelo config cluster
Select the distribution and enter the NiFi details when asked during the configuration.

✔ Is the 'Database Service' up and running? [y_n]: : n ✔ Standalone_Ambari_CM ✔ Nifi Enter Your Cluster's Display Name: : ad_nifi Enter the cluster name to use (MUST be all lowercase & unique): : ad_nifi Enter the cluster name to use (MUST be all lowercase & unique): : ad_nifi INFO: Creating Post dirs. ✔ Enter the hosts file path for Nifi (MUST formatted, one IP_host per line): : _data01_acceldata_nifihosts Discovered NIFI Hosts: path for Nifi (MUST formatted, one IP_host per line): : _data01_acceld ✓ nifi1000.sre.axl.iti:8888 ✓ nifi1001.sre.axl.iti:8888 ✓ nifi1002.sre.axl.iti:8888 Would you like to continue with the above NIFI nodes? [y_n]: : Y ✓ acceldata.conf file generated successfully. INFO: Creating post config files INFO: Writing the .dist files INFO: Clustername : ad_nifi INFO: Performing PreCheck of Files INFO: Setting the active cluster Creating hydra inventory SSH Key Algorithm used (RSA_DSA)?: : RSA Which user should connect over SSH: : root SSH private key file path for connecting to hosts: : _data01_clusterkey plat01.dev.axl.iti is the IP address of the Pulse Server, Is this correct? [Y_N]: : Y Would you like to enable NTP Stats? [y_n]: : y ✔ Would you like to setup LogSearch? [y_n]: : Y ? Select the logs for components that are installed_enabled in your target cluster: nifi, kern, syslog ✓ Generated the vars.yml file successfully Configuring notifications ✓ Generated the notifications.yml file successfully INFO: Please run 'accelo deploy core' to deploy APM core using this configuration.

Last updated on

Was this page helpful?