Redis Sentinel as Celery Broker for Airflow

This document describes how to configure Redis Sentinel as the Celery broker for Apache Airflow. Redis Sentinel provides high availability by managing automatic failover between Redis master and replica nodes.

Prerequisites

Before configuring Airflow, ensure the following Redis Sentinel setup is already in place:

• 3 Sentinel nodes
• 1 Redis master
• 2 Redis replicas

Redis Sentinel Connection URL

Airflow connects to Redis Sentinel using a Sentinel-formatted broker URL. The connection string must include all Sentinel hosts and the Redis database number.

Example Broker URL Format

sentinel:__<sentinel-1>:26379_0;

sentinel:__<sentinel-2>:26379_0;

sentinel:__<sentinel-3>:26379_0

Where:

• sentinel-x represents the Sentinel hostnames
• 26379 is the default Sentinel port
• 0 is the Redis database number

Broker URL Override Configuration

Because the Redis Sentinel URL format is more complex than standard Redis configurations, an advanced configuration option has been introduced:

Broker URL Override (Advanced)

• This field accepts the Redis Sentinel URL in the format shown above.
• When specified, it overrides all other Celery broker configurations.
• If left empty, Airflow uses the default broker configuration defined earlier.

Additional Required Configuration for Redis Sentinel

When using Redis Sentinel with Celery, the Sentinel master name must also be configured.

Add the following entry to the Custom airflow-celery_broker_transport_options-site configuration:

master_name=mymaster

Replace mymaster with the actual master name defined in your Redis Sentinel setup.