Sidecar (Rolling) Upgrade from Kafka 2 to Kafka 3

This document describes the step-by-step process for upgrading an existing Kafka 2.x cluster to Kafka 3.x using a sidecar (rolling) upgrade approach — where some brokers temporarily run Kafka 2 while others run Kafka 3.

This ensures no cluster downtime and enables a safe, gradual migration.

1. Current Setup

The existing Kafka 2 cluster is running with the following data-log directory:

2. Validate the Existing Kafka 2 Setup

Before starting the upgrade, verify that the Kafka 2 cluster is fully functional.

List Topics

xxxxxxxxxx

[root@ce7-f1-node01 bin]# ./kafka-topics.sh --list \

  --bootstrap-server localhost:port \

  --command-config client-kerb.prop

Create a Sample Topic

xxxxxxxxxx

[root@ce7-f1-node01 bin]# ./kafka-topics.sh --create \

  --topic dilraj \

  --bootstrap-server localhost:port \

  --command-config client-kerb.prop \

  --partitions 3 \

  --replication-factor 3

Created topic dilraj.

Produce Sample Data

xxxxxxxxxx

[root@ce7-f1-node01 bin]# ./kafka-console-producer.sh \

  --topic dilraj \

  --bootstrap-server localhost:port \

  --producer.config client-kerb.prop

>This message was sent when Kafka was running in v2.8.2

Consume the Data

xxxxxxxxxx

[root@ce7-f1-node01 bin]# ./kafka-console-consumer.sh \

  --topic dilraj \

  --bootstrap-server localhost:port \

  --consumer.config client-kerb.prop \

  --from-beginning

This message was sent when Kafka was running in v2.8.2

Verification: Messages are successfully produced and consumed. Kafka 2 setup is healthy.

3. Prepare for Upgrade

Update Broker Protocol Version

In the Ambari UI → Kafka → Configs → Custom kafka-broker, add or update:

kafka2.version

Perform a rolling restart of all Kafka 2 brokers to apply this configuration. This ensures backward compatibility between Kafka 2 and Kafka 3 brokers during the upgrade.

4. Install Kafka 3 Management Pack (MPack)

On all nodes on which Kafka 2 brokers were already running, install the Kafka 3 MPack:

xxxxxxxxxx

[root@odin01 ~]# ambari-server install-mpack \

  --mpack=ambari-mpacks-kafka3-3.7.1.3.3.6.0-1.tar.gz

Expected Output:

xxxxxxxxxx

Ambari Server 'install-mpack' completed successfully.

Then restart the Ambari Server:

xxxxxxxxxx

[root@odin01 ~]# ambari-server restart

5. Perform the Sidecar Upgrade

In the sidecar approach, brokers are upgraded one at a time — some running Kafka 2, others Kafka 3, until all are upgraded.

Stop One Kafka 2 Broker

Using Ambari, stop the Kafka 2 broker on one node.

Add Kafka 3 Service

In Ambari UI, add the Kafka 3 service. During the wizard, select only one host — the same host where the stopped Kafka 2 broker was running.

Update Kafka 3 Configuration

Before starting the Kafka 3 broker, make the following adjustments:

Ensure all configurations under Advanced kafka3-broker match the Kafka 2 settings.

Start Kafka 3 Broker

Start the new Kafka 3 broker via Ambari.

Once verified that the broker joins the cluster and the state is healthy, proceed to the next step.

Remove the Corresponding Kafka 2 Broker

In Ambari, delete the Kafka 2 broker (that was stopped earlier) from the Kafka 2 service.

At this point:

• n-1 brokers are running Kafka 2
• 1 broker is running Kafka 3

The mixed-version cluster (Kafka 2 + Kafka 3) is functional.

6. Verify Post-Upgrade Functionality

Test the end-to-end functionality again:

xxxxxxxxxx

[root@odin01 bin]# ./kafka-console-consumer.sh \

  --topic dilraj \

  --bootstrap-server 10.100.11.51:6667 \

  --consumer.config client-kerb.prop \

  --from-beginning

This message was sent when kafka was running in v2.8.2

Monitor logs and metrics to confirm no errors or partition reassignments are pending.

7. Repeat for Remaining Brokers

Repeat steps 5.1 → 5.5 for each remaining Kafka 2 broker until all are replaced by Kafka 3 brokers.

8. Finalize the Upgrade

Once all brokers are running Kafka 3 and the cluster is stable:

1. Update the protocol version under Custom kafka3-broker:

xxxxxxxxxx

inter.broker.protocol.version=3.7.1.3.3.6.0-1

2. Restart brokers one by one to apply the new protocol version.

9. Cleanup and Post-Upgrade Notes

• Once all brokers have been migrated and validated:

  • Delete the Kafka 2 service from Ambari.
  • Do not reinstall Kafka 2 using default configurations, as it may corrupt or override existing data directories.

Kafka 3 Upgrade Complete

Your Kafka cluster is now fully upgraded to Kafka 3.x with all brokers running on the new version, reusing existing data directories and ZooKeeper quorum.