Ambari High Availability Setup

This document outlines the process for configuring high availability (HA) for Ambari Server using an active-passive setup. It also includes instructions for optional load balancing of agents with HAProxy.

Prerequisites

1. Two identically configured Ambari server instances.
2. A shared external database for both Ambari server instances.
3. A load balancer (such as HAProxy).

Steps to Setup

Perform the following steps:

1. Set Up Ambari Servers

• Deploy two Ambari server instances.
• Ensure both instances have identical configurations, including Ambari property files.
• Connect both servers to an external database to maintain consistency.
• Use ambari-server setup command to configure both instances in the same manner.

2. Install and Configure HAProxy

• If HAProxy is not installed, use sudo yum install haproxy -y.
• Modify the HAProxy configuration file at /etc/haproxy/haproxy.cfg.
• Replace existing frontend and backend settings with:

frontend ambari_frontend

    bind *:8080

    mode http

    default_backend ambari_backend

​

backend ambari_backend

    mode http

    balance roundrobin

    option httpchk GET /

    server ambari_server1 <Ambari_Server_1_IP>:8080 check

    server ambari_server2 <Ambari_Server_2_IP>:8080 backup check

In this configuration:

• ambari_server1 is the primary server.
• ambari_server2 serves as the backup, receiving traffic only if ambari_server1 is down.
• The backup designation indicates ambari_server2 is a passive server.

3. Restart HAProxy

Apply the new settings by restarting HAProxy:

sudo systemctl restart haproxy

sudo systemctl enable haproxy

sudo systemctl status haproxy

4. Optional: Configure HAProxy Logging

• Set up logging by installing rsyslog and configuring logging for HAProxy:

sudo yum install rsyslog -y

echo '$ModLoad imudp' | sudo tee /etc/rsyslog.d/haproxy.conf

echo '$UDPServerRun 514' | sudo tee -a /etc/rsyslog.d/haproxy.conf

echo '$template HaproxyLog,"/var/log/haproxy.log"' | sudo tee -a /etc/rsyslog.d/haproxy.conf

echo ':programname, startswith, "haproxy" -?HaproxyLog' | sudo tee -a /etc/rsyslog.d/haproxy.conf

sudo systemctl restart rsyslog

• Check the logs at /var/log/haproxy.log to confirm proper logging.

5. Update Ambari Agent Configuration

• Direct Ambari agents to connect to the active server using the provided script:

#!/bin/bash

​

# Ambari Server Details

AMBARISERVER=$(hostname -f)

USER=admin

PASSWORD=admin

PORT=8080

PROTOCOL=http

​

# Load Balancer Details

ACTIVE_AMBARI=odp1-mar2024.adsre.com

​

# Get the cluster name

CLUSTER=$(curl -ksu $USER:$PASSWORD -i -H 'X-Requested-By: ambari' $PROTOCOL://$AMBARISERVER:$PORT/api/v1/clusters | grep -o '"cluster_name" : "[^"]*' | awk -F'"' '{print $4}')

​

# Get the list of all hosts in the cluster

HOSTS=($(curl -su $USER:$PASSWORD $PROTOCOL://$AMBARISERVER:$PORT/api/v1/clusters/$CLUSTER/hosts | grep -o '"host_name" : "[^"]*' | awk -F'"' '{print $4}'))

​

# Loop through each host and update ambari-agent.ini

for HOST in "${HOSTS[@]}"; do

   # Get current hostname from ambari-agent.ini on remote host

   CURRENT_HOSTNAME=$(ssh $HOST "grep 'hostname=' /etc/ambari-agent/conf/ambari-agent.ini | cut -d'=' -f2")

​

   # Check if the hostname needs to be updated

   if [ "$CURRENT_HOSTNAME" != "$ACTIVE_AMBARI" ]; then

       # Backup the original ambari-agent.ini file

       ssh $HOST "sudo cp -f /etc/ambari-agent/conf/ambari-agent.ini /etc/ambari-agent/conf/ambari-agent.ini.bak"

​

       # Update hostname in ambari-agent.ini on remote host

       ssh $HOST "sed -i 's/^hostname=.*/hostname=$ACTIVE_AMBARI/' /etc/ambari-agent/conf/ambari-agent.ini"

​

       # Restart ambari-agent on remote host

       ssh $HOST "sudo systemctl restart ambari-agent"

       echo "Hostname on $HOST updated to $ACTIVE_AMBARI. Ambari agent restarted."

   else

       echo "Hostname on $HOST is already set to $ACTIVE_AMBARI. No action needed."

   fi

done

Note If Ambari Server is unavailable, use the script to redirect agents to the operational server.

Conclusion

Following these steps ensures the successful implementation of high availability for the Ambari Server with HAProxy, enhancing both availability and reliability of your managed cluster.

Known Limitations

1. Heartbeat Lost After Failover: If the first Ambari Server goes down, HAProxy will redirect the request to the second Ambari Server, but you may observe "Heartbeat Lost" in the Ambari UI. Use 5. Update Ambari Agent Configuration (Step 5) to address any heartbeat issues post-failover.
2. Access Denied for Database User: Modify database permissions to allow access from the HA host if you see the following error:

Caused by: java.sql.SQLException: Access denied``for``user 'ambari'@'ambari-ha.adsre.com' (using password: YES)

Solution:

grant all privileges on *.* to 'ambari'@'ambari-ha.adsre.com' indentified by 'bigdata' with grant option ;

3. No Database Selected: Verify that server.jdbc.url in ambari.properties specifies the correct database name, if you see the following error:

Caused by: java.sql.SQLException: No database selected

Solution:

server.jdbc.url=jdbc:mysql://<database-hostname>:3306/ambari