Kafka MirrorMaker

MirrorMaker 2.0 (MM2) simplifies the mirroring or replication of topics between Kafka clusters using the Kafka Connect framework for easier configuration and scaling. It dynamically detects topic changes and synchronizes source and target topic properties, including offsets and partitions.

Cluster-to-Cluster Mirroring: MM2 replicates topics between Kafka clusters, useful for disaster recovery, data aggregation, and cloud migration.

Active-Active Replication: MM2 supports bidirectional replication, allowing both clusters to produce and consume records while MM2 manages synchronization.

Topic Configuration Synchronization: MM2 replicates topic configurations, ensuring that target cluster topics match the source configurations.

Consumer Group Offset Synchronization: It synchronizes consumer offsets between clusters, allowing consumers to continue from their last read point when switching clusters.

Prerequisites:

  • An environment with at least two Kafka clusters.
  • Kafka version higher than 2.4.x.
  • The source cluster should have data points and topics to test various MM2 replication features.

Required Deployment Configuration

The connect-mirror-maker.sh script, bundled with the Kafka library, implements a distributed MirrorMaker 2 (MM2) cluster, managing the Connect workers internally based on a configuration file. The MirrorMaker driver internally creates and handles pairs of connectors: MirrorSourceConnector, MirrorSinkConnector, MirrorCheckpointConnector, and MirrorHeartbeatConnector.

Adding JMX Configurations

Add JMX configurations in the connect-mirror-maker.sh script before the exec statement:

Bash
Copy

Example port: 9998

Export KAFKA_JMX_OPTS in connect-mirror-maker.sh file:

Bash
Copy

For Kerberos-enabled clusters, the JAAS configuration must be exported to KAFKA_OPTS or specified in the MM2 configuration file:

Bash
Copy

Adding mm2.properties File

Place the mm2.properties file under path_to_kafka/config.

Example mm2.properties file:

Bash
Copy

Starting MirrorMaker 2

Start MirrorMaker 2 by specifying the mm2.properties file:

Bash
Copy

Once you have completed the server configuration, you must answer a few questions to view Kafka MM2 on the UI. Follow the prompts to confirm monitoring, provide the Kafka MirrorMaker node details, verify the hostnames, and enter the JMX port. Here’s an example of the questions you will encounter:

Bash
Copy

Kafka Mirror Maker on Pulse UI

To view the Kafka Mirror Maker dashboard, navigate to the left navigation menu and select Kafka > MirrorMaker. The dashboard will be displayed. Use the global calendar to select a time period (e.g., Today, Last 12 hours, Last 3 Months) or choose a custom time period and click Apply to view the charts with data for that period. Additionally, use the HostName filter to display data for selected host names only.

The following charts are displayed on the Kafka MirrorMaker dashboard:

NameDescription
Source -> DestinationDisplays the source and destination clusters involved in the mirroring process.
Last RebalanceShows the time elapsed since the last rebalance of the Kafka clusters.
Completed RebalanceIndicates the total number of rebalances completed.
Throughput/SecMeasures the throughput in messages per second.
Topics SyncedShows the number of topics that have been synchronized between clusters.
Bytes TransferredDisplays the total amount of data transferred in bytes.
Replication LatencyIndicates the latency in milliseconds for replication between the source and target clusters.
Message CountA line graph showing the message count over time with legends for cluster details.
Topics Throughput/SecA line chart displaying the throughput per second for topics over time.
Topics Bytes TransferredA table listing nodes, synced topics, and bytes transferred for each topic.
Topics Replication LatencyA table showing nodes, synced topics, and the replication latency for each topic.
Task Error MetricsA table with columns for connector, task, total record errors, failures, and retries.
Synced TopicsA table listing nodes and their corresponding topics that have been synchronized.
Source Task MetricsA table with columns for connector, task, source active record count, poll total, and write total.
Connector Task MetricsA table including connector, task, offset commit success/failure percentages, and other metrics.
Connect Worker MetricsA table displaying metrics like total tasks, destroyed tasks, paused tasks, restarting tasks, etc.
Type to search, ESC to discard
Type to search, ESC to discard
Type to search, ESC to discard
Next to read:
Kafka Cruise Control

For additional help, contact www.acceldata.force.com OR call our service desk +1 844 9433282

Copyright © 2025

On This Page
Kafka MirrorMakerPrerequisites:Required Deployment ConfigurationAdding JMX ConfigurationsAdding mm2.properties FileStarting MirrorMaker 2Kafka Mirror Maker on Pulse UI