In-Place Upgrade from Kafka 2 to Kafka 3

This document outlines the steps to perform an in-place upgrade of the Kafka cluster from Kafka 2.x to Kafka 3.x using the same data directories and ZooKeeper quorum.

Pre-requisites

• The existing Kafka 2.x cluster is running and stable.
• The current data log directory (used by Kafka 2) is available and will be reused by Kafka 3.
• Ensure Ambari is up and running.
• Upgrade will be performed in ZooKeeper mode (not KRaft).

1. Validate the Existing Kafka 2 Setup

Before starting the upgrade, verify that the Kafka 2 cluster is fully functional.

List Topics

[root@ce7-f1-node01 bin]# ./kafka-topics.sh --list \

  --bootstrap-server localhost:port \

  --command-config client-kerb.prop

Create a Sample Topic

[root@ce7-f1-node01 bin]# ./kafka-topics.sh --create \

  --topic dilraj \

  --bootstrap-server localhost:port \

  --command-config client-kerb.prop \

  --partitions 3 \

  --replication-factor 3

Created topic dilraj.

Produce Sample Data

[root@ce7-f1-node01 bin]# ./kafka-console-producer.sh \

  --topic dilraj \

  --bootstrap-server localhost:port \

  --producer.config client-kerb.prop

>This message was sent when Kafka was running in v2.8.2

Consume the Data

[root@ce7-f1-node01 bin]# ./kafka-console-consumer.sh \

  --topic dilraj \

  --bootstrap-server localhost:port \

  --consumer.config client-kerb.prop \

  --from-beginning

This message was sent when Kafka was running in v2.8.2

Verification: Messages are successfully produced and consumed. Kafka 2 setup is healthy.

2. Prepare for the Upgrade

Update Broker Protocol Compatibility

Before installing Kafka 3, update the following property under Ambari → Kafka → Configs → Custom kafka-broker:

Restart all Kafka brokers from Ambari to apply the configuration. This ensures smooth compatibility during the upgrade.

3. Install Kafka 3 Management Pack (MPack)

Run the following commands on all the nodes on which Kafka 2 is installed:

[root@ce7-f1-node01 ODP-3.2.3.3-2-MPACKs]# ambari-server install-mpack \

  --mpack=ambari-mpacks-kafka3-3.7.1.3.2.3.3-2.tar.gz

Then restart the Ambari Server:

[root@ce7-f1-node01 ODP-3.2.3.3-2-MPACKs]# ambari-server restart

4. Upgrade to Kafka 3

Stop Kafka 2 Brokers

Stop all Kafka 2 brokers via Ambari.

Install Kafka 3

From Ambari, add the Kafka 3 service and perform the following configurations before starting brokers:

Mandatory Notes:

• The upgrade must be performed in ZooKeeper mode.
• Ensure Kafka 3 uses the same log directory as Kafka 2.
• Ensure ZooKeeper quorum configuration matches Kafka 2.

Update Configuration in Ambari

Under Advanced kafka3-broker, configure the following:

5. Validate Post-Upgrade Functionality

After starting all Kafka 3 brokers:

Verify Topics and Data

Try consuming the earlier created topic:

[root@ce7-f1-node01 bin]# ./kafka-console-consumer.sh \

  --topic dilraj \

  --bootstrap-server localhost:port \

  --consumer.config client-kerb.prop \

  --from-beginning

This message was sent when Kafka was running in v2.8.2

Expected: You should be able to consume messages produced before the upgrade.

6. Finalize the Upgrade

Once the Kafka 3 cluster is verified to be stable:

1. Update inter.broker.protocol.version under Custom kafka3-broker:

inter.broker.protocol.version={kafka3.version}

2. Restart brokers one by one to apply the new protocol version.

7. Clean Up

• Delete the old Kafka 2 service from Ambari.
• Do not reinstall Kafka with default configurations — this may overwrite or corrupt existing data.

8 Rollback (If Required)

If the Kafka 3 upgrade encounters issues, limited rollback options are available. However, rollback feasibility depends on the upgrade stage and whether broker protocol changes were applied.

Rollback Feasibility Matrix

Safe Rollback Procedure (Before Protocol Bump)

If rollback is still feasible (i.e., protocol version not yet updated):

1. Stop all Kafka 3 brokers via Ambari.
2. Remove Kafka 3 service from Ambari (do not delete data directories).
3. Reinstall Kafka 2 MPack and service using Ambari.
4. Ensure log.dirs and zookeeper.connect point to the same values as before.
5. Start Kafka 2 brokers and validate cluster health by listing and consuming topics.

Verification: You should be able to consume previously stored messages.

When Rollback is Not Possible

• Rollback is not supported once
• The Kafka3 cluster has run with a newer inter.broker.protocol.version.
• Any log compaction or retention cleanup occurred after the upgrade.

At this stage, the data format has changed, and Kafka 2.x brokers cannot interpret Kafka 3 logs safely.

Upgrade Complete

You now have a fully functional Kafka 3 cluster using the existing Kafka 2 data directories and ZooKeeper quorum.