Scheduler and Webserver High Availability for Airflow

This document describes how to configure High Availability (HA) for Apache Airflow Schedulers and Webservers in an ODP deployment. Scheduler HA improves reliability and fault tolerance, while Webserver HA ensures consistent UI access through a load balancer.

High Availability Support in ODP Airflow

ODP Airflow supports running multiple schedulers and webservers in an HA configuration.

  • During a fresh installation, users can choose to add multiple schedulers and webservers.

  • For an existing installation, additional scheduler or webserver components can be added via Ambari → Hosts → Add, and assigning the required roles.

Additional Configurations for Scheduler HA

The following configurations are recommended to ensure consistent and predictable behavior when running multiple schedulers.

Configure these settings in Advanced airflow-scheduler-site.

Required and Recommended Properties

In Advanced airflow-scheduler-site

use_job_schedule: True

  • Mandatory for Scheduler HA.

  • Enables consistent job scheduling across multiple schedulers.

deactivate_unknown_dags: False

  • Prevents schedulers from disabling DAGs managed by other schedulers.

parsing_processes: 4

  • Improves DAG parsing throughput through parallelism

scheduler_health_check_threshold: 90

  • Defines an HA-friendly scheduler liveness threshold (in seconds).

For Webserver HA, deploying a Load Balancer in front of Airflow webservers is strongly recommended.

Ambari may not enforce all required configurations automatically in HA setups; therefore, the following settings should be verified and applied manually if necessary.

Webserver Configuration Updates

Update the following settings in Advanced airflow-webserver-site.

enable_proxy_fix: True

  • Required when Airflow is deployed behind a load balancer.

  • Ensures proper handling of X-Forwarded-* HTTP headers.

Load Balancer URL Configuration

Update the Load Balancer URL in the following locations.

In Advanced airflow-webserver-site

base_url: <Load Balancer URL for Airflow UI>

In Custom airflow-webserver-site

web_server_url: <Load Balancer URL for Airflow UI>

These values must point to the Load Balancer endpoint, not individual web server hosts.

HAProxy Setup for Airflow Webservers (Load Balancer)

The following example demonstrates configuring HAProxy as a load balancer for Airflow web servers on RHEL 8. Similar steps can be applied to Ubuntu or other Linux distributions.

Install HAProxy

yum install haproxy

HAProxy Configuration Example

Update /etc/haproxy/haproxy.cfg as shown below. Replace hostnames and ports with values from your Airflow deployment.

global log _dev_log local0 log _dev_log local1 notice maxconn 50000 daemon user haproxy group haproxy defaults log global mode http option httplog option dontlognull timeout connect 5s timeout client 50s timeout server 50s frontend http_front bind *:8080 mode http default_backend airflow_web_backends backend airflow_web_backends mode http balance roundrobin option httpchk GET _health server web1 poc1.acceldata.ce:8889 check server web2 poc2.acceldata.ce:8889 check
Info

Replace poc1.acceldata.ce and poc2.acceldata.ce with the actual hostnames and ports of your Airflow webservers.

Start HAProxy

sudo systemctl start haproxy


  Last updated on Jan 28, 2026
Next to read:
Basic Authentication for Flower UI in Airflow
null
On This Page
Scheduler and Webserver High Availability for AirflowHigh Availability Support in ODP AirflowAdditional Configurations for Scheduler HAAdditional Steps and Configurations for Webserver HAHAProxy Setup for Airflow Webservers (Load Balancer)Required and Recommended PropertiesWebserver Configuration UpdatesLoad Balancer URL ConfigurationInstall HAProxyHAProxy Configuration ExampleStart HAProxy