Configure Pulse to Monitor Multiple Kafka clusters under a single Hadoop Cluster

This document is created under the assumption that a Hadoop cluster having multiple kafka cluster, each node will only be a part of single kafka cluster.

Steps

Add Multiple Kafka Clusters in Config

Edit:

Bash
    
xxxxxxxxxx
 
<acceldata_clustername>.conf
Copy

Under kafka.connectors, define multiple clusters:

Bash
    
xxxxxxxxxx
 
kafka.connectors = [  {    name = "kafka-0.11-connector"    trigger = "scheduled"    service = "kafka"    maxFailRetries = 5     clusters = [      {        name = "kafka-01"        servers = "host1:9092,host2:9092,host3:9092"        zk_servers = "host1:2181,host2:2181,host3:2181/kafka"        securityProtocol = "SASL_PLAINTEXT"      },      {        name = "kafka-02"        servers = "host4:9092"        zk_servers = "host1:2181,host2:2181,host3:2181/kafka12"        securityProtocol = "SASL_PLAINTEXT"      }    ]  }]
Copy

Key Notes

Keep the first cluster name the same as the Hadoop cluster name (avoids reconfiguring all agents).
Ensure:
- Unique name per Kafka cluster
- Correct zk_servers (different chroot if shared ZK)

Push Config to Database

Bash
    
xxxxxxxxxx
 
accelo admin database push-config
Copy

Restart Kafka Connector

Bash
    
xxxxxxxxxx
 
accelo restart ad-kafka-connector
Copy

Define Target Hosts for Reconfiguration

Edit/create:

Bash
    
xxxxxxxxxx
 
$ACCELDATA_HOME/work/<cluster_name>/hydra_hosts_override.yml
Copy

Example:

YAML
    
xxxxxxxxxx
 
hosts:  add:    - host1    - host2    - host3  remove:    - host4    - host5    - host6
Copy

Purpose:

add → nodes to be reassigned to the new Kafka cluster
remove → excluded nodes

Update JMX Cluster Mapping

Edit:

Bash
    
xxxxxxxxxx
 
override.yml
Copy

Bash
    
xxxxxxxxxx
 
static:  jmx_rmi_db: <kafka-cluster-name>_jmx
Copy

Example:

Bash
    
xxxxxxxxxx
 
kafka-02_jmx
Copy

Reconfigure Agents

Bash
    
xxxxxxxxxx
 
accelo reconfig cluster
Copy

This updates:

Kafka JMX cluster name
Only for hosts in add list

Repeat for Additional Kafka Clusters

Repeat Steps 4–6 for each Kafka cluster.

Validation Checklist

Check the db_name in /opt/pulse/jmx/config/enabled/vm_kafka.json file.
Check the cluster name in JMX metrics through Dashplot studio.

Last updated on

Was this page helpful?