Configure HDP Kafka and Kafka 3

This page describes how to configure Kafka and Kafka 3 in HDP clusters so Pulse can collect metrics, monitor topics, and track job activity.

Set JMX Port (Kafka 2.x)

In the Ambari UI:

Add the following parameters at the end of the file.
Add the following parameter at the end of the file to set the JMX port without any security.

Bash
    
 
export JMX_PORT=${JMX_PORT:-9999}
Copy

Set JMX port (Kafka 3.x)

In the Ambari UI:

Navigate to Kafka > Configs > Advanced kafka3-env > kafka3-env.
Add the following parameter at the end of the file to set the JMX port without any security.

Kafka 3 with ZooKeeper

Bash
    
 
export JMX_PORT=${JMX_PORT:-8987}
Copy

Kafka 3 with KRaft

Bash
    
 
export JMX_PORT=${JMX_PORT:-8988}
Copy

Enable Basic Authentication (Optional)

In kafka-env template or kafka3-env, choose and add one of the following JMX configurations based on your security requirements.

Enable Basic Authentication on JMX Remote Port (Optional)

To enable basic authentication on the JMX remote port, add the following parameters:

Bash
    
export KAFKA_JMX_OPTS="$KAFKA_JMX_OPTS -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.access.file=</path/to/jmxremote.access/file> -Dcom.sun.management.jmxremote.password.file=</path/to/jmxremote.password/file>"
Copy

Enable TLS/SSL on JMX Remote Port (Optional)

To enable TLS/SSL on the JMX Remote port, add the following parameters:

Bash
    
export KAFKA_JMX_OPTS="$KAFKA_JMX_OPTS -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.access.file=</path/to/jmxremote.access/file> -Dcom.sun.management.jmxremote.password.file=</path/to/jmxremote.password/file> -Dcom.sun.management.jmxremote.ssl=true -Dcom.sun.management.jmxremote.registry.ssl=true -Djavax.net.ssl.keyStore=</path/to/keystore.jks/file> -Djavax.net.ssl.keyStorePassword=<Keystore Password> -Djavax.net.ssl.trustStore=</path/to/truststore.jks/file> -Djavax.net.ssl.trustStorePassword=<Truststore Password>"
Copy

Configure Kafka ACLs for HDFS User

Run the following commands as the Kafka user to grant All operations on all topics and groups:

Bash
    
# Grant topic permissions ./kafka-acls.sh   --bootstrap-server <broker ip>   --command-config client-kerb.prop   --add   --allow-principal User:hdfs   --allow-host '*'   --operation All  --topic '*'
Copy

Bash
    
# Grant group permissions  ./kafka-acls.sh   --bootstrap-server <broker ip>   --command-config client-kerb.prop   --add   --allow-principal User:hdfs   --allow-host '*'   --operation All --group '*'
Copy

Result

Kafka and Kafka 3 expose JMX metrics on the configured ports.
Optional authentication and TLS/SSL secure the connection.
ACLs ensure Pulse can read topic and consumer group metadata without manual intervention.

Last updated on

Was this page helpful?