Ambari High Availability Setup

This document outlines the process for configuring high availability (HA) for Ambari Server using an active-passive setup. It also includes instructions for optional load balancing of agents with HAProxy.

Prerequisites

Two identically configured Ambari server instances.
A shared external database for both Ambari server instances.
A load balancer (such as HAProxy).

Steps to Setup

Perform the following steps:

1. Set Up Ambari Servers

Deploy two Ambari server instances.
Ensure both instances have identical configurations, including Ambari property files.
Connect both servers to an external database to maintain consistency.
Use ambari-server setup command to configure both instances in the same manner.

2. Install and Configure HAProxy

If HAProxy is not installed, use sudo yum install haproxy -y.
Modify the HAProxy configuration file at /etc/haproxy/haproxy.cfg.
Replace existing frontend and backend settings with:

frontend ambari_frontend bind *:8080 mode http default_backend ambari_backend backend ambari_backend mode http balance roundrobin option httpchk GET / server ambari_server1 <Ambari_Server_1_IP>:8080 check server ambari_server2 <Ambari_Server_2_IP>:8080 backup check

In this configuration:

ambari_server1 is the primary server.
ambari_server2 serves as the backup, receiving traffic only if ambari_server1 is down.
The backup designation indicates ambari_server2 is a passive server.

3. Restart HAProxy

Apply the new settings by restarting HAProxy:

sudo systemctl restart haproxy sudo systemctl enable haproxy sudo systemctl status haproxy

4. Optional: Configure HAProxy Logging

Set up logging by installing rsyslog and configuring logging for HAProxy:

sudo yum install rsyslog -y echo '$ModLoad imudp' | sudo tee /etc/rsyslog.d/haproxy.conf echo '$UDPServerRun 514' | sudo tee -a /etc/rsyslog.d/haproxy.conf echo '$template HaproxyLog,"/var/log/haproxy.log"' | sudo tee -a /etc/rsyslog.d/haproxy.conf echo ':programname, startswith, "haproxy" -?HaproxyLog' | sudo tee -a /etc/rsyslog.d/haproxy.conf sudo systemctl restart rsyslog

Check the logs at /var/log/haproxy.log to confirm proper logging.

5. Update Ambari Agent Configuration

Direct Ambari agents to connect to the active server using the provided script:

#!/bin/bash # Ambari Server Details AMBARISERVER=$(hostname -f) USER=admin PASSWORD=admin PORT=8080 PROTOCOL=http # Load Balancer Details ACTIVE_AMBARI=odp1-mar2024.adsre.com # Get the cluster name CLUSTER=$(curl -ksu $USER:$PASSWORD -i -H 'X-Requested-By: ambari' $PROTOCOL://$AMBARISERVER:$PORT/api/v1/clusters | grep -o '"cluster_name" : "[^"]*' | awk -F'"' '{print $4}') # Get the list of all hosts in the cluster HOSTS=($(curl -su $USER:$PASSWORD $PROTOCOL://$AMBARISERVER:$PORT/api/v1/clusters/$CLUSTER/hosts | grep -o '"host_name" : "[^"]*' | awk -F'"' '{print $4}')) # Loop through each host and update ambari-agent.ini for HOST in "${HOSTS[@]}"; do # Get current hostname from ambari-agent.ini on remote host CURRENT_HOSTNAME=$(ssh $HOST "grep 'hostname=' /etc/ambari-agent/conf/ambari-agent.ini | cut -d'=' -f2") # Check if the hostname needs to be updated if [ "$CURRENT_HOSTNAME" != "$ACTIVE_AMBARI" ]; then # Backup the original ambari-agent.ini file ssh $HOST "sudo cp -f /etc/ambari-agent/conf/ambari-agent.ini /etc/ambari-agent/conf/ambari-agent.ini.bak" # Update hostname in ambari-agent.ini on remote host ssh $HOST "sed -i 's/^hostname=.*/hostname=$ACTIVE_AMBARI/' /etc/ambari-agent/conf/ambari-agent.ini" # Restart ambari-agent on remote host ssh $HOST "sudo systemctl restart ambari-agent" echo "Hostname on $HOST updated to $ACTIVE_AMBARI. Ambari agent restarted." else echo "Hostname on $HOST is already set to $ACTIVE_AMBARI. No action needed." fi done

Note If Ambari Server is unavailable, use the script to redirect agents to the operational server.

Conclusion

Following these steps ensures the successful implementation of high availability for the Ambari Server with HAProxy, enhancing both availability and reliability of your managed cluster.

Known Limitations

Heartbeat Lost After Failover: If the first Ambari Server goes down, HAProxy will redirect the request to the second Ambari Server, but you may observe "Heartbeat Lost" in the Ambari UI. Use 5. Update Ambari Agent Configuration (Step 5) to address any heartbeat issues post-failover.
Access Denied for Database User: Modify database permissions to allow access from the HA host if you see the following error:

Caused by: java.sql.SQLException: Access denied``for``user 'ambari'@'ambari-ha.adsre.com' (using password: YES)

Solution:

grant all privileges on *.* to 'ambari'@'ambari-ha.adsre.com' indentified by 'bigdata' with grant option ;

No Database Selected: Verify that server.jdbc.url in ambari.properties specifies the correct database name, if you see the following error:

Caused by: java.sql.SQLException: No database selected

Solution:

server.jdbc.url=jdbc:mysql://<database-hostname>:3306/ambari

Last updated on Apr 18, 2024

Was this page helpful?