Deploy Standalone Kafka 4

This page explains how to deploy a standalone Kafka 4 cluster in KRaft mode: get the Kafka binaries, configure the controller and broker, initialize storage, start the cluster, and create, write, and read events from a topic.

Get Kafka

Download the standalone binaries of Kafka 4.3.0 and extract them:

Start the Kafka environment

Kafka4 requires Java to run. Ensure that Java 17 is installed by running the following command:

Set JAVA_HOME to point to Java 17. Kafka 4.3.0 only runs in KRaft mode. To get started, follow the steps below.

Kafka with KRaft

KRaft mode is Kafka's newer architecture that eliminates the dependency on ZooKeeper. In KRaft mode, Kafka manages metadata natively using its Raft-based consensus protocol.

Step 1: Configure the Kafka controllers

Controllers in KRaft mode are responsible for managing metadata across the cluster.

Edit the config/kraft-controller.properties file to define your controller configurations:

Step 2: Configure the Kafka broker

Once the controllers are set up, configure the brokers, which handle message storage and processing.

Edit the config/broker.properties file to define your broker configurations:

Step 3: Initialize the Kafka storage

In KRaft mode, Kafka uses a unique cluster.id to manage metadata. Before you start any services, generate and initialize the storage directories for the Kafka controllers and brokers:

Step 4: Start the Kafka controllers

Once the configuration is complete, start the Kafka controller services. In KRaft mode, start the controllers first, because they manage the metadata for the brokers.

Step 5: Start the broker

Open another terminal session and run the broker service:

Create a topic to store your events

A topic can be thought of as a folder in a file system, and individual events as files stored within that folder.

Before you can produce and consume events, create a topic. Open a new terminal and run the following command:

Kafka's command-line tools offer several options for various operations. Run the kafka-topics.sh command without arguments to get a list of these options. For example, to get details about the newly created topic, such as its partition count, run:

Write events into the topic

A Kafka client interacts with the Kafka brokers over the network to write or read events. When the brokers receive events, they store them in a durable, fault-tolerant way, ensuring the data remains available for as long as necessary — even indefinitely, if configured.

To write events to your Kafka topic, use the Kafka console producer. Each line of input you provide is written as a separate event to the specified topic.

Run the following command to start the console producer client and write events:

Start typing messages, for example:

Each line is treated as an individual event and sent to the quickstart-events topic. To stop the producer client, press Ctrl-C at any time.

Read the events

To read the events you've written to your Kafka topic, use the Kafka console consumer. This client connects to the Kafka brokers, retrieves the events, and displays them in the terminal.

Open a new terminal session and run the following command to start the console consumer:

The console outputs the events you previously produced:

As with the producer, you can stop the consumer client by pressing Ctrl-C at any time.

Terminate the Kafka environment

Once you've reached the end of the quickstart, you can tear down the Kafka environment, or continue exploring it.

1. Stop the producer and consumer clients with Ctrl-C, if you haven't done so already.

2. Stop the KRaft broker and controller with Ctrl-C.

If you also want to delete the data from your local Kafka environment, including any events you created, run: