Single Node Standalone Cluster Guide (Kafka/Druid/Nifi)

KAFKA

Configure the Pulse Core Component for Standalone Kafka Cluster

Pulse supports only Kafka versions 1.0.0 and higher.

Run the following command to get started:

Bash
    
 
accelo config cluster
Copy

The Acceldata CLI asks you for information about your environment. The following table lists the questions asked by Acceldata CLI and a guide to help you answer the questions.

Questions asked by CLI	Guidelines for answering the questions
Is this current node an 'Edge Node' of the cluster? [y/n]:	Type y if the node on which you are installing is a Edge Node. Else type n.
Which distribution do you use?	Select Standalone
Stand-alone configuration selected:	Displays the configuration selected.

The CLI asks you for few more information to generate the configuration file.

Questions asked by CLI	Guidelines for answering the questions
Enter Your Cluster’s Name:	Enter the name of the cluster.
Enter Your Cluster’s Display Name:	Enter the name you want for the cluster to display
Select the components you would like to install a) MEMSQL b) KAFKA	Select the components that you want to install. You can use the arrows to move, space to select, and type in to filter
Is your Kafka version greater than '0.11.0'? [y/n]	Type y if the Kafka version is greater than V0.11.0.
Enter one of the Kafka bootstrap server's URI	Enter the Kafka boot server URL. Example: host1.kf.com:6667
What security do you use? a. None b. Plan/SSL c. Kerberos	Select your security type.
Do you use TLS ? a. No b. Yes	Select Yes if you use TLS or else select NO.
Detected Kafka brokers:	Based on your responses CLI would detect the Kafka brokers.
Is the above the information correct? [y/n]:	Type y to confirm if the detected Kafka brokers are correct.
Enter the Zookeeper Server URLs (comma separated with http/https & port):	Enter the Zookeeper server URL and use comma separate http, https, and the port.
Enter Kafka's Zookeeper Chroot Name (enter '/' for root):	Enter the Kafka Zookeeper roots
Would you like to continue with the above configuration? [y/n]:	Type y if you want to continue with the configuration or else type n.
Would you like to enable LDAP? [y/n]:	Type y if you want to enable LDAP or else type n.
Is Kerberos enabled in this cluster? [y/n]:	Enter your Kerberos Keytab username (Must have required HDFS permissions)
Enter the cluster name to use (MUST be all lowercase & unique):	Enter the cluster name to be used. Ensure that the name should be in all lowercase and unique.

You will see the following message.

Bash
    
INFO: Trying to generate host-roles-map file ...  ✓ SuccessINFO: Trying to generate alert-endpoints file ...INFO: Trying to generate FSAnalytics scripts ...WARN: Cannot find any HDFS installations from this cluster. Skipping the FS Analytics configuration.INFO: Edit the config files 'ad-core-connectors.yaml' and 'ad-fsanalyticsv2-connector.yaml'ERROR: Cannot find the required hadoop conf directory at the path '/etc/hadoop/conf'IMPORTANT: Please make sure this directory exists at the path '/etc/hadoop/conf' and contains the files 'core-site.xml' and 'hdfs-site.xml'INFO: Please run 'accelo deploy core' to deploy APM core using this configuration.
Copy

You may be asked to answer the following questions if Kerberos is enabled.

Questions asked by CLI	Guidelines for answering the questions
Would you like to continue configuring Kerberos? [y/n]:	Type y if you want to continue configuring the Kerberos
Authentication realm	Add the authentication realm. Example: AZ.PULSE.COM
KDC address	Add the KDC address. Example: host1.kdc.com:88
Kerberos principal	Add the Kerberos principal. Example: admin/admin@AZ.PULSE.COM
KDC admin server address	Add the KDC administration server address. Example: host1.kdcadmin.com:749

Configure Kafka when Security Is Disabled

If security is not enabled in your standalone Kafka or Kafka 3 cluster, update the acceldata.conf and ad-kafka-connector.yml configuration files as described below.

Update the acceldata.conf file.

Update consumerSecurityProtocol and securityProtocol:

From "SASL_PLAINTEXT"
To "PLAINTEXT".

Bash
    
 
enableConsumerGroupCache = truebatchSize = 5zk_secure = falseconsumerSecurityProtocol = "PLAINTEXT"filterConsumerGroups = ""filterConsumerGroupsState = ""securityProtocol = "PLAINTEXT"SASLEnabled = "false"TLSEnabled = "false"
Copy

Create the ad-kafka-connector.yml file (if not present).

Run the following command to generate the configuration file:

Bash
    
 
accelo admin makeconfig ad-kafka-connector
Copy

Update the ad-kafka-connector.yml file.

In the $AcceloHome/config/docker/addons/ad-kafka-connector.yml file, set KERBEROS_ENABLED to false.

Bash
    
 
version: "2"services:  ad-kafka-connector:    image: ad-kafka-connector    container_name: ""    environment:    - MONGO_URI=<URI>    - MONGO_ENCRYPTED=true    - MONGO_SECRET=<Secret>    - KERBEROS_ENABLED=false    - OTEL_JAVAAGENT_ENABLED=false    volumes:    - /etc/localtime:/etc/localtime:ro    - /etc/hosts:/etc/hosts:ro    ulimits: {}    ports: []    depends_on: []    opts: {}    restart: ""    extra_hosts: []    network_alias: []label: Kafka Connector
Copy

Checking the Configuration

Open the files located in the address /data01/acceldata/config/acceldata<CLUSTER_NAME>.conf_ to verify if all the configurations are correct.

DRUID (HDP or Standalone Cluster)

SSH to all druid hosts and create a directory druid-accel-extensions under /usr/hdp/<cluster_version>/druid/extensions
Make following changes in druid advanced configs in Ambari UI

Add druid-accel-extensions in config druid.extensions.loadList=["druid-accel-extensions"...
Update or add druid.emitter=acceldata / example for any custom composing emitters druid.emitter.composing.emitters=["ambari-metrics","kafka","acceldata"]
druid.monitoring.emissionPeriod=PT15S
druid.emitter.acceldata.hostname=<pulse_host>
druid.emitter.acceldata.port=19009
druid.emitter.acceldata.databaseName=<cluster_name>
druid.emitter.acceldata.adStreamingUrl=http://{{pulse_host}}:19005
druid.emitter.acceldata.dbUser=<db-user>
druid.emitter.acceldata.dbPassword=<password>
druid.emitter.acceldata.group=<cluster_name>
druid.emitter.acceldata.monitor=druid
druid.emitter.acceldata.maxQueueSize=500

Add the monitors that you want to monitor in the Druid Monitors. Ensure to add service-specific monitors in service specific-configs as mentioned in the following configurations. Prepare a separate config groups in Ambari for common(default), broker, and historical.

Common Configs:

druid.monitoring.monitors=["io.druid.server.metrics.EventReceiverFirehoseMonitor","io.druid.java.util.metrics.JvmThreadsMonitor","io.druid.java.util.metrics.JvmCpuMonitor","io.druid.client.cache.CacheMonitor","io.druid.java.util.metrics.SysMonitor","io.druid.java.util.metrics.JvmMonitor"]

Broker Configs:

druid.monitoring.monitors=["io.druid.client.cache.CacheMonitor","io.druid.java.util.metrics.SysMonitor","io.druid.java.util.metrics.JvmMonitor","io.druid.java.util.metrics.JvmCpuMonitor","io.druid.java.util.metrics.JvmThreadsMonitor","io.druid.server.metrics.QueryCountStatsMonitor"]

Historical Configs:

druid.monitoring.monitors=["io.druid.client.cache.CacheMonitor", "io.druid.java.util.metrics.SysMonitor", "io.druid.java.util.metrics.JvmMonitor", "io.druid.java.util.metrics.JvmCpuMonitor", "io.druid.java.util.metrics.JvmThreadsMonitor","io.druid.server.metrics.HistoricalMetricsMonitor"]

Restart the affected components after the configuration changes.
Check on the Influx DB for new measurements getting added and also validate the service logs on the Druid for any errors after the restart activity.

NiFi

Configuring the Apache NiFi

Follow the steps to setup the Apache NiFi in Pulse:

Obtain the ad-nifi-1.0.nar download link from Acceldata team using the provided credentials
Add the NiFi NAR files to all the NiFi daemons in the location Nifi library classpath. example : /usr/hdf/<hdf_version>/nifi/lib

Check for NiFi library location where AD NAR files needs to be placed.

Restart the NiFi Services.
Login to NiFi UI, navigate to the AccelDataReportingTask in the controller settings.

Set the properties into the fields.

Field	Description
Monitor Group Name	Enter the name of cluster name as defined during Pulse installation
AccelData Influx Metric Collector URL	Add the VictoriaMetrics insert URL of the AccelData Nifi Metric Collector
Elasticsearch Concurrent Threads	Set the number. The number of threads to push the data to Elasticsearch
Elasticsearch Batch Size	Set the Elasticsearch Batch size per Thread
Max Provenance Events	Set the number to send a trigger when the maximum events occurs
Influx Batch Size	Set the number of Influx Batch size
Influx Batch Flush Duration	Set the duration of the influx batch flush in milliseconds.

Checking the Configuration

Check to see if the configuration is working as expected by following the steps:

Check if the VictoriaMetrics is updated with the prefix <groupname>_nifi.
The Elastic has the index with the name as Group Name + '-nifi-'.

Pulse Installation

Prerequisite

Create a Nifihosts file. The file's content will be as follows:

Each line of the file will have a Nifi host and port.

Example:

Bash
    
 
nifihosts1:9090nifihosts2:9090
Copy

If the port is not given, the default port is used. For example, 9090 for alerts endpoints usage.

Configure the Pulse Core Component

Run accelo config cluster

Bash
    
 
✔ Is the 'Database Service' up and running? [y/n]: : n█✔ Stand-Alone✔ NifiEnter Your Cluster's Display Name: : ad_nifi█Enter the cluster name to use (MUST be all lowercase & unique): : ad_nifi█Enter the cluster name to use (MUST be all lowercase & unique): : ad_nifi█INFO: Creating Post dirs.✔ Enter the hosts file path for Nifi (MUST formatted, one IP/host per line): : /data01/acceldata/nifihosts█Discovered NIFI Hosts: path for Nifi (MUST formatted, one IP/host per line): : /data01/acceld    ✓  nifi1000.sre.axl.iti:8888    ✓  nifi1001.sre.axl.iti:8888    ✓  nifi1002.sre.axl.iti:8888Would you like to continue with the above NIFI nodes? [y/n]: : Y█✓ acceldata.conf file generated successfully.INFO: Creating post config filesINFO: Writing the .dist filesINFO: Clustername : ad_nifiINFO: Performing PreCheck of FilesINFO: Setting the active clusterCreating hydra inventorySSH Key Algorithm used (RSA/DSA)?: : RSA█Which user should connect over SSH: : root█SSH private key file path for connecting to hosts: : /data01/clusterkey█plat01.dev.axl.iti is the IP address of the Pulse Server, Is this correct? [Y/N]: : Y█Would you like to enable NTP Stats? [y/n]: : y█✔ Would you like to setup LogSearch? [y/n]: : Y█? Select the logs for components that are installed/enabled in your target cluster:  nifi, kern, syslog✓ Generated the vars.yml file successfullyConfiguring notifications✓ Generated the notifications.yml file successfullyINFO: Please run 'accelo deploy core' to deploy APM core using this configuration.
Copy

Last updated on

Was this page helpful?