Acceldata
 Pulse
Search documentation… ⌘K

Last updated: Sep 26, 2026 20:14 UTC

Configure HDP Kafka and Kafka 3

This page describes how to configure Kafka and Kafka 3 in HDP clusters so Pulse can collect metrics, monitor topics, and track job activity.

Set JMX Port (Kafka 2.x)

In the Ambari UI:

  • Add the following parameters at the end of the file.
  • Add the following parameter at the end of the file to set the JMX port without any security.
export JMX_PORT=${JMX_PORT:-9999}

Set JMX port (Kafka 3.x)

In the Ambari UI:

  • Navigate to Kafka > Configs > Advanced kafka3-env > kafka3-env.
  • Add the following parameter at the end of the file to set the JMX port without any security.

Kafka 3 with ZooKeeper

export JMX_PORT=${JMX_PORT:-8987}

Kafka 3 with KRaft

export JMX_PORT=${JMX_PORT:-8988}

Enable Basic Authentication (Optional)

In kafka-env template or kafka3-env, choose and add one of the following JMX configurations based on your security requirements.

Enable Basic Authentication on JMX Remote Port (Optional)

To enable basic authentication on the JMX remote port, add the following parameters:

export KAFKA_JMX_OPTS="$KAFKA_JMX_OPTS -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.access.file=</path/to/jmxremote.access/file> -Dcom.sun.management.jmxremote.password.file=</path/to/jmxremote.password/file>"

Enable TLS/SSL on JMX Remote Port (Optional)

To enable TLS/SSL on the JMX Remote port, add the following parameters:

export KAFKA_JMX_OPTS="$KAFKA_JMX_OPTS -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.access.file=</path/to/jmxremote.access/file> -Dcom.sun.management.jmxremote.password.file=</path/to/jmxremote.password/file> -Dcom.sun.management.jmxremote.ssl=true -Dcom.sun.management.jmxremote.registry.ssl=true -Djavax.net.ssl.keyStore=</path/to/keystore.jks/file> -Djavax.net.ssl.keyStorePassword=<Keystore Password> -Djavax.net.ssl.trustStore=</path/to/truststore.jks/file> -Djavax.net.ssl.trustStorePassword=<Truststore Password>"

Configure Kafka ACLs for HDFS User

Run the following commands as the Kafka user to grant All operations on all topics and groups:

# Grant topic permissions 
./kafka-acls.sh --bootstrap-server <broker ip> --command-config client-kerb.prop --add --allow-principal User:hdfs --allow-host '*' --operation All --topic '*'

# Grant group permissions  
./kafka-acls.sh --bootstrap-server <broker ip> --command-config client-kerb.prop --add --allow-principal User:hdfs --allow-host '*' --operation All --group '*'

Result

  • Kafka and Kafka 3 expose JMX metrics on the configured ports.
  • Optional authentication and TLS/SSL secure the connection.
  • ACLs ensure Pulse can read topic and consumer group metadata without manual intervention.