Remove Kudu Masters (Ambari Multi-Master)

This page describes how to remove a Kudu master from an Ambari-managed multi-master deployment.

Ambari Service Component Management

Stop and Delete the Kudu Master Service

Before making configuration changes, ensure that the master you want to remove is properly stopped in Ambari:

1. Navigate to the Hosts page in the Ambari Web UI.
2. Click the host containing the Kudu Master component (for example, kudu3.acceldata.dvl).
3. In the Components section, locate the Kudu Master component.
4. Stop the component if it is still running.
5. Delete the component using the Delete option from the component drop-down menu.

Master Removal Command Execution

Run the kudu master remove command to remove the master from the cluster configuration.

kudu master remove <existing master comma separated list> <new_master_hostname>

Example:

kudu master remove <existing master comma separated list> <new_master_hostname>

This command removes kudu3.acceldata.dvl from the cluster configuration. The remaining masters (kudu1.acceldata.dvl and kudu2.acceldata.dvl) now form the new master quorum.

Sample output:

I20250613 03:18:51.246625 1906910 tool_action_master.cc:566] Successfully removed master kudu3.acceldata.dvl:7051 from the cluster. Please follow the next steps from the Kudu administration documentation on "Removing Kudu Masters from a Multi-Master Deployment" or "Recovering from a dead Kudu Master in a Multi-Master Deployment" as appropriate.

WAL and Data Directory Cleanup

Remove the Write-Ahead Log (WAL) and data directories from the removed master to prevent it from rejoining the cluster.

rm -rf /var/lib/kudu/wal/

This step is especially important if a corrupted master is being removed. Allowing a corrupted master to rejoin the cluster can cause data loss.

Service Restart Sequence

Ambari automatically updates the master_addresses and tserver_master_addrs configuration parameters for the remaining masters and tablet servers.

Restart the remaining masters one by one:

1. In Ambari, go to Kudu > Summary.
2. Restart the kudu1.acceldata.dvl master first.
3. Wait for it to show as healthy.
4. Restart the kudu2.acceldata.dvl master.

Restart all tablet servers:

• Restart all tablet servers at once, or use Ambari’s Rolling Restart option.
• In Ambari, go to Kudu > Summary.
• Select all tablet servers.
• Click Restart to apply the updated master configuration.

Verification Steps

Verify the cluster state by accessing the master web interface.

1. Open the web UI for each remaining master (http://<hostname>:8051/masters).
2. Verify that only the remaining masters are listed.
3. Confirm that one master is shown as LEADER and the other as FOLLOWER.

Run a Cluster Health Check

Run the following command to ensure that all tablet servers are properly connected to the updated master configuration:

kudu cluster ksck kudu1.acceldata.dvl:7051,kudu2.acceldata.dvl:7051