Configure Pulse to Monitor Multiple Kafka clusters under a single Hadoop Cluster

Info

This document is created under the assumption that a Hadoop cluster having multiple kafka cluster, each node will only be a part of single kafka cluster.

Steps

Add Multiple Kafka Clusters in Config

Edit:

<acceldata_clustername>.conf

Under kafka.connectors, define multiple clusters:

kafka.connectors = [ { name = "kafka-0.11-connector" trigger = "scheduled" service = "kafka" maxFailRetries = 5 clusters = [ { name = "kafka-01" servers = "host1:9092,host2:9092,host3:9092" zk_servers = "host1:2181,host2:2181,host3:2181/kafka" securityProtocol = "SASL_PLAINTEXT" }, { name = "kafka-02" servers = "host4:9092" zk_servers = "host1:2181,host2:2181,host3:2181/kafka12" securityProtocol = "SASL_PLAINTEXT" } ] } ]

Key Notes

Keep the first cluster name the same as the Hadoop cluster name (avoids reconfiguring all agents).
Ensure:
- Unique name per Kafka cluster
- Correct zk_servers (different chroot if shared ZK)

Push Config to Database

accelo admin database push-config

Restart Kafka Connector

accelo restart ad-kafka-connector

Define Target Hosts for Reconfiguration

Edit/create:

$ACCELDATA_HOME/work/<cluster_name>/hydra_hosts_override.yml

Example:

hosts: add: - host1 - host2 - host3 remove: - host4 - host5 - host6

Purpose:

add → nodes to be reassigned to the new Kafka cluster
remove → excluded nodes

Update JMX Cluster Mapping

Edit:

override.yml

static: jmx_rmi_db: <kafka-cluster-name>_jmx

Example:

kafka-02_jmx

Reconfigure Agents

accelo reconfig cluster

This updates:

Kafka JMX cluster name
Only for hosts in add list

Repeat for Additional Kafka Clusters

Repeat Steps 4–6 for each Kafka cluster.

Validation Checklist

Check the db_name in /opt/pulse/jmx/config/enabled/vm_kafka.json file.
Check the cluster name in JMX metrics through Dashplot studio.

Last updated on

Was this page helpful?