Acceldata
Pulse

Last updated: Sep 26, 2026 21:12 UTC

Configure Pulse to Monitor Multiple Kafka clusters under a single Hadoop Cluster

Info

This document is created under the assumption that a Hadoop cluster having multiple kafka cluster, each node will only be a part of single kafka cluster.

Steps

  • Add Multiple Kafka Clusters in Config

Edit:

<acceldata_clustername>.conf

Under kafka.connectors, define multiple clusters:

kafka.connectors = [
  {
    name = "kafka-0.11-connector"
    trigger = "scheduled"
    service = "kafka"
    maxFailRetries = 5
    clusters = [
      {
        name = "kafka-01"
        servers = "host1:9092,host2:9092,host3:9092"
        zk_servers = "host1:2181,host2:2181,host3:2181/kafka"
        securityProtocol = "SASL_PLAINTEXT"
      },
      {
        name = "kafka-02"
        servers = "host4:9092"
        zk_servers = "host1:2181,host2:2181,host3:2181/kafka12"
        securityProtocol = "SASL_PLAINTEXT"
      }
    ]
  }
]

Key Notes

  • Keep the first cluster name the same as the Hadoop cluster name (avoids reconfiguring all agents).
  • Ensure:
    • Unique name per Kafka cluster
    • Correct zk_servers (different chroot if shared ZK)

  • Push Config to Database
accelo admin database push-config

  • Restart Kafka Connector
accelo restart ad-kafka-connector

  • Define Target Hosts for Reconfiguration

Edit/create:

$ACCELDATA_HOME/work/<cluster_name>/hydra_hosts_override.yml

Example:

hosts:
  add:
    - host1
    - host2
    - host3
  remove:
    - host4
    - host5
    - host6

Purpose:

  • add → nodes to be reassigned to the new Kafka cluster
  • remove → excluded nodes

  • Update JMX Cluster Mapping

Edit:

override.yml
static:
  jmx_rmi_db: <kafka-cluster-name>_jmx

Example:

kafka-02_jmx

  • Reconfigure Agents
accelo reconfig cluster

This updates:

  • Kafka JMX cluster name
  • Only for hosts in add list

  • Repeat for Additional Kafka Clusters

Repeat Steps 4–6 for each Kafka cluster.


Validation Checklist

  • Check the db_name in /opt/pulse/jmx/config/enabled/vm_kafka.json file.
  • Check the cluster name in JMX metrics through Dashplot studio.