Kafka 3

Kafka 3 is a distributed event streaming platform designed to handle real-time data feeds with enhanced scalability, performance, and resilience. It introduces new features like KRaft mode and provides a more flexible configuration to improve broker management and fault tolerance. Additionally, Kafka3 focuses on improving replication, monitoring, and security features to support large-scale, enterprise-level event streaming use cases.

Key Concepts

Kafka3 includes several core components that enhance its ability to deliver distributed real-time event streaming:

• Producer: Sends messages to Kafka topics by writing them to specific partitions. It supports batching and compression to optimize performance.
• Consumer: Reads data from Kafka topics, supporting different consumption models such as at-least-once, at-most-once, and exactly-once delivery semantics.
• Broker: Kafka brokers act as intermediaries that store messages in topic partitions and replicate them to ensure fault tolerance. Kafka 3.7.1 introduces improvements in broker scalability and resource management.
• Controller: Responsible for managing partition leadership and handling cluster metadata. With Kafka3 KRaft Mode, controller functionality is embedded within brokers, eliminating the need for ZooKeeper.

KRaft Mode: Kafka3 introduces KRaft (Kafka Raft Metadata Mode), which allows Kafka to manage metadata without relying on ZooKeeper. This simplifies operations, improves fault tolerance, and enhances scalability by embedding the metadata controller directly within the brokers.

Installing Kafka 3 using Ambari

Perform the following steps to install the Ambari Kafka 3 Mpack:

1. Download the branch from the code and zip it with the tar.gz extension.
2. Upload the gzipped file to the ambari-server.
3. Execute the following command to install Kafka 3.

ambari-server install-mpack --mpack=kafka3-3.7.1.tar.gz --verbose

Upon running the above command, the following message is displayed.

INFO: Management pack kafka3-3.7.1.tar.gz successfully installed! Please restart ambari-server.

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

Ambari Server 'install-mpack' completed successfully.

Restart the ambari-server using the following command:

ambari-server restart

4. Log in to the Ambari UI and navigate to the Add Kafka3 service.
5. Select the hosts for Kafka 3 brokers.

4. Under Advanced kraft-broker-controller, enable the Kraft configuration as this is very crucial for installation:

The above configuration must be set based on your deployment type:

• Set Enable KRaft to false for the ZooKeeper-based deployment.
• Set Enable KRaft to true for the KRaft mode deployments.

Zookeeper based Deployment

1. Set Enable KRaft to false under Advanced kraft-broker-controller. Apart from this configuration, do not set any kraft related configurations through Ambari.

2. Set Log directories[log.dirs] under Advanced kafka3-broker ,which can be used to store Kafka 3 log data. By default /kafka3-logs is used as the log directory.

2. Under Advanced kafka3-env , you can configure Kafka log directory to store Kafka 3 operational logs. Additionally, you can specify Kafka PID directory, which is used to store the process ID (PID) files for Kafka 3.

4. Specify the port on which you want your kafka-servers to listen.

5. By default Kafka 3 uses the /kafka3 zookeeper namespace to avoid conflicts with existing Kafka clusters by specifying a chroot path in the connection string.

KRaft mode deployment

1. Set Enable KRaft to true under Advanced kraft-broker-controller.

2. In the kraft mode of deployment, you can select certain hosts as controller, broker, or both depending on the size of the cluster and use case.

Broker Role

• A broker is sometimes called a node or server, is responsible for orchestrating the storage transmission of messages within the Kafka cluster.

Controller Role

• A controller coordinates the cluster, managing and tracking its metadata. Controllers form a metadata quorum, where each one serves either as an active controller or a hot standby for the current active controller.
• While Kafka nodes can serve both as brokers and controllers, it is sometimes preferable to separate these functions, especially in more complex deployments. In simpler setups, combining the broker and controller roles can lead to greater efficiency.
• To maintain high availability, a majority of controllers must be operational. For instance, with 3 controllers, the cluster can tolerate 1 failure; with 5 controllers, it can handle up to 2 failures.

Specify a list of comma separated node.id@hostname which you want to select as controllers and brokers in kraft-controller-list and kraft-broker-list respectively under Advanced kraft-broker-controller.

For example, in this case there are three Kafka brokers and one controller.

3. Set log directories [log.dirs] under Advanced kraft-broker and Advanced kraft-controller which can used store kraft-broker and kraft-controller log data respectively. By default /kraft-broker-logs and /kraft-controller-logs are used as the log directories for broker and controller respectively.

4. Under Advanced kraft-broker-env and Advanced kraft-controller-env , you can configure the Kraft-broker log directory and Kraft-controller log directory to store kafka3-kraft operational logs. Additionally, you can specify Kafka PID directory for broker and controller which is used to store the process ID (PID) files for kraft-broker and kraft-controller.

Advanced kraft-controller-env

5. Specify the port on which you want your kraft-brokers and kraft-controllers to listen under Advanced kraft-broker and Advanced kraft-controller respectively.

Advanced kraft-controller

If the Ranger-Kafka plugin is enabled in Kraft mode, the following policy must be added via Ranger UI:

Create a new policy Kraft Cluster Metadata Policy,which must include _cluster_metadata topic.

6. You can also modify the remaining conigs depending on your use case for controller and broker under Advanced kraft-controller,Advanced kraft-broker, Advanced kraft-controller-env ,and Advanced kraft-controller-env.
7. Start kafka-3 in kraft mode.

8. When you use KRaft instead of ZooKeeper, you must use current, non-deprecated, and configurations settings. The settings to use are described in the following table.

Note: If the Ranger-Kafka plugin is enabled, make sure to use rangerlookup keytab and principal. You can edit these configurations through the Ranger UI under Service Manager > Edit Service > Config Properties.