Configure Pulse to Monitor Multiple Kafka clusters under a single Hadoop Cluster

Steps

• Add Multiple Kafka Clusters in Config

Edit:

<acceldata_clustername>.conf

Under kafka.connectors, define multiple clusters:

kafka.connectors = [
 {
 name = "kafka-0.11-connector"
 trigger = "scheduled"
 service = "kafka"
 maxFailRetries = 5
 clusters = [
 {
 name = "kafka-01"
 servers = "host1:9092,host2:9092,host3:9092"
 zk_servers = "host1:2181,host2:2181,host3:2181/kafka"
 securityProtocol = "SASL_PLAINTEXT"
 },
 {
 name = "kafka-02"
 servers = "host4:9092"
 zk_servers = "host1:2181,host2:2181,host3:2181/kafka12"
 securityProtocol = "SASL_PLAINTEXT"
 }
 ]
 }
]

Key Notes

• Keep the first cluster name the same as the Hadoop cluster name (avoids reconfiguring all agents).
• Ensure:

Unique name per Kafka cluster

- Correct zk_servers (different chroot if shared ZK)

• ---

  Push Config to Database

accelo admin database push-config

• ---

  Restart Kafka Connector

accelo restart ad-kafka-connector

• ---

  Define Target Hosts for Reconfiguration

Edit/create:

$ACCELDATA_HOME/work/<cluster_name>/hydra_hosts_override.yml

Example:

hosts:
 add:
 - host1
 - host2
 - host3
 remove:
 - host4
 - host5
 - host6

Purpose:

• add → nodes to be reassigned to the new Kafka cluster
• remove → excluded nodes

• ---

  Update JMX Cluster Mapping

Edit:

override.yml

static:
 jmx_rmi_db: <kafka-cluster-name>_jmx

Example:

kafka-02_jmx

• ---

  Reconfigure Agents

accelo reconfig cluster

This updates:

• Kafka JMX cluster name
• Only for hosts in add list

• ---

  Repeat for Additional Kafka Clusters

Repeat Steps 4–6 for each Kafka cluster.

---

Validation Checklist

• Check the db_name in /opt/pulse/jmx/config/enabled/vm_kafka.json file.
• Check the cluster name in JMX metrics through Dashplot studio.