Configure Pulse to Monitor Standalone Kafka 3
This page describes how to configure Pulse to monitor a standalone Kafka 3 deployment, supporting both KRaft and ZooKeeper modes. This configuration enables real-time observability and health monitoring of Kafka components.
Run Accelo Config Command
On a Pulse node, run the following command.
accelo config clusterAnswer the CLI Questions
Upon running the accelo config cluster command, Pulse CLI prompts with the following questions.
| Questions asked by CLI | Guidelines for answering the questions |
|---|---|
| Select the cluster type | Select the cluster type as Standalone. [x] Stand-Alone [ ] Ambari [ ] Cloudera |
| Select the Kafka 3 Standalone component. | [ ] Kafka [ ] NiFi [ ] Spark [x] Kafka 3 If your Kafka 3 deployment uses KRaft mode, select Kafka 3. If your Kafka 3 deployment uses ZooKeeper mode, select Kafka. Based on your environment, select the appropriate component and enter the required details. In this document, the scenario assumes a Kafka 3 deployment with KRaft mode. However, the remaining steps are the same for both modes. |
| Enter Your Cluster's Display Name | Enter the cluster name. |
| Enter Kafka 3 Version | Enter the Kafka 3 version. |
| Enter one of the Kafka bootstrap server's URIs (e.g., host1.kf.com:6667): | Enter one of the Kafka 3 bootstrap server URIs |
| Select the detected Kafka 3 brokers after Pulse auto-detects them. | ✓ Detected kafka3 brokers: kafka101.acceldata.dvl:9092 kafka102.acceldata.dvl:9092 kafka103.acceldata.dvl:9092 ✓ Total number of brokers: 3 |
| Select the security type used by your Kafka cluster. | ✓ None Plain/SASL Kerberos Scram Based on the selection of each security type, the CLI prompts with a set of questions to enter the details. In this document, the security type selected is None. |
| Enter the Kraft Controller URLs (comma separated with host & port, e.g., krafthost1:19092,krafthost2:19092) | Enter the Kraft Controller URLs (e.g., krafthost1:19092,krafthost2:19092). |
Do you want to monitor KafkaMM2 [y/n] Enter the Kafka MirrorMaker nodes (comma-separated hosts, e.g., mm2kafka1,mm2kafka2) | Type y if you want to monitor Kafka Mirror Maker 2 and enter the host names. |
Do you want to monitor Kafka Connect [y/n] Enter the Kafka Connect nodes (comma-separated hosts, e.g., kafkaconnect1,kafkaconnect2): | Type y if you want to monitor Kafka Connect and enter the host names. |
Do you want to monitor Kafka Cruise Control [y/n] Enter the Kafka Cruise Control nodes (comma-separated hosts, e.g., kafkaconnect1,kafkaconnect2) | Type y if you want to monitor Kafka Cruise Control and enter the host names. |
Do you want to monitor Schema Registry [y/n] Enter the Schema Registry nodes (comma-separated hosts, e.g., schemaregistry1,schemaregistry2) | Type y if you want to monitor the Schema Registry and enter the host names. |
| Add JMX ports for data collection | Enter the Kafka3 Broker Port: : 9092 Enter the Kafka JMX Port (Ex, 9999): 9999 Enter the Kafka3 MirrorMaker JMX Port (Ex, 9998): 9998 Enter the Cruise Control3 JMX Port (default: 56667): 56667 Enter the Kafka3 Connect JMX Port (default: 9994): 9994 Enter the Schema Registry JMX Port (default: 8996)::8996 |
| Select logs for installed/enabled components. | Select the logs for components that are installed/enabled in your target cluster: [ x ] kraftbroker_err [ x ] kafka3_connect_err, kern [ x ] kafka3_controller [ x ] kafka3_connect [ x ] kafka3_cruise_control [ x ] registry_err [ x ] kafka3_log_cleaner [ x ] kafka3_server [ x ] kafka3_server_gc [ x ] kraftcontroller_err [ x ] syslog, registry_server [ x ] kafka3_state_change [ x ] kafka3_cruise_control_err |
Add the Component Log Directory
Add the component log directory by creating an override.yml file (if not exist) in $AcceloHome/work/<clustername>/ and specifying the required configuration settings as below.
log_locations: kafka3_connect: - path: /path_to/connect.log type: DEFAULT kafka3_connect_err: - path: /path_to/kafka-connect.err type: KAFKA_ERR kafka3_controller: - path: /path_to/controller.log type: DEFAULT kafka3_cruise_control: - path: /path_to/kafkacruisecontrol.log type: DEFAULT kafka3_cruise_control_err: - path: /path_to/cruise-control.err type: KAFKA_ERR kafka3_log_cleaner: - path: /path_to/log-cleaner.log type: DEFAULT kafka3_server: - path: /path_to/server.log type: DEFAULT kafka3_server_gc: - path: /path_to/kafkaServer-gc.log type: DEFAULT kafka3_state_change: - path: /path_to/state-change.log type: DEFAULT kraftbroker_err: - path: /path_to/kraft-broker.err type: DEFAULT kraftcontroller_err: - path: /path_to/kraft-controller.err type: DEFAULT registry_err: - path: /path_to/registry.err type: REGISTRY registry_server: - path: /path_to/registry.log type: REGISTRYRun the following command to update the vars.yml file with the new configuration.
accelo reconfig clusterVerify the Configuration File
Once the configuration is complete, you can verify the final setup in the following path.
$AcceloHome/config/acceldata<cluster name>.confRun Accelo Deploy Core Command
Run the following command. Once the deployment is complete, the Pulse UI will be up and running.
accelo deploy core