Configure ODP Kafka and Kafka 3

This page describes how to enable JMX for Kafka so that Pulse can collect metrics.

You can optionally secure JMX with basic authentication or TLS. You must also grant the required ACLs to ensure Pulse has access.

Set JMX Port (Kafka 2.x)

In the Ambari UI:

• Navigate to Kafka > Configs > Advanced kafka-env > kafka-env template.
• Add the following parameter at the end of the file to set the JMX port without any security.

export JMX_PORT=${JMX_PORT:-9999}

Set JMX port (Kafka 3.x)

In the Ambari UI:

1. Navigate to Kafka > Configs > Advanced kafka3-env > kafka3-env template.
2. Add the following parameter at the end of the file to set the JMX port without any security.

Kafka 3 with ZooKeeper

export JMX_PORT=${JMX_PORT:-8987}

Kafka 3 with KRaft

export JMX_PORT=${JMX_PORT:-8988}

Enable Authentication on JMX Remote Port

In kafka-env template or kafka3-env, choose and add one of the following JMX configurations based on your security requirements.

Enable Basic Authentication on JMX Remote Port (Optional)

To enable basic authentication on the JMX remote port, add the following parameters:

export KAFKA_JMX_OPTS="$KAFKA_JMX_OPTS -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.access.file=</path/to/jmxremote.access/file> -Dcom.sun.management.jmxremote.password.file=</path/to/jmxremote.password/file>"

Enable TLS/SSL on JMX Remote Port (Optional)

To enable TLS/SSL authentication on the JMX remote port, add the following parameters:

export KAFKA_JMX_OPTS="$KAFKA_JMX_OPTS -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.access.file=</path/to/jmxremote.access/file> -Dcom.sun.management.jmxremote.password.file=</path/to/jmxremote.password/file> -Dcom.sun.management.jmxremote.ssl=true -Dcom.sun.management.jmxremote.registry.ssl=true -Djavax.net.ssl.keyStore=</path/to/keystore.jks/file> -Djavax.net.ssl.keyStorePassword=<Keystore Password> -Djavax.net.ssl.trustStore=</path/to/truststore.jks/file> -Djavax.net.ssl.trustStorePassword=<Truststore Password>"

Set Kafka ACLs for Pulse (HDFS user example)

Run the following commands as the Kafka user to grant All operations on all topics and groups:

Grant topic permissions:

# Grant topic permissions 

./kafka-acls.sh   --bootstrap-server <broker ip>   --command-config client-kerb.prop   --add   --allow-principal User:hdfs   --allow-host '*'   --operation All  --topic '*'

Grant group permissions:

# Grant group permissions  

./kafka-acls.sh   --bootstrap-server <broker ip>   --command-config client-kerb.prop   --add   --allow-principal User:hdfs   --allow-host '*'   --operation All --group '*'

Result

• Kafka exposes JMX metrics on the configured port.
• JMX can be secured with Basic Auth and/or TLS.
• Pulse has Kafka ACLs to read topics and groups.