Single Node Installation

RHEL 8/9 Setup

Prerequisites

To establish the necessary development tools and Python packages, ensure the installation of Python packages with versions 3.11 and above. Note that while this documentation employs Python 3.11, Apache Airflow is compatible with Python versions starting from 3.11 and higher.

Database Setup

For an optimal test drive experience of Airflow, choose a robust database backend such as PostgreSQL or MySQL. By default, Airflow uses SQLite, which is mainly intended for development.

Airflow supports only specific versions of database engines. Verify that your database version is compatible as older versions might not support certain SQL features:

• PostgreSQL: Versions 12, 13, 14, 15, 16

• MySQL: Version 8.0, Innovation

• MSSQL (experimental, support ending in version 2.9.0): 2017, 2019

• SQLite: Version 3.15.0 or later

Before installing Apache Airflow, set up a compatible database. Choose PostgreSQL or MySQL based on your needs and ensure that your database version meets the following minimum requirements:

• MySQL: Version 8.0 or higher

• PostgreSQL: Version 12 or higher

NOTE Apache Airflow does not support Oracle databases.

Refer: Setup a Database Backend-Apache Airflow

Refer to the guidelines below for your selected database to prepare and configure it for Apache Airflow.

PostgreSQL Database Setup

To integrate PostgreSQL with Apache Airflow, complete the following steps to install and configure it:

Install the psycopg2-binary Python package:

Install PostgreSQL:

1. Install the repository RPM:

2. Disable the built-in PostgreSQL module:

3. Install PostgreSQL:

Initialize and start PostgreSQL:

1. Access the PostgreSQL shell:

2. Inside the PostgreSQL shell, execute the following commands:

• Create the Airflow database:

• Create the Airflow user with a password:

• Set client encoding, default transaction isolation, and timezone for the Airflow user:

• Grant all privileges on the Airflow database to the Airflow user:

• Exit the PostgreSQL shell:

Configure PostgreSQL settings for Airflow:

1. Open the PostgreSQL configuration file:

2. Inside the file, modify the following settings:

• Change and uncomment the listen_addresses to '*':

• Uncomment the following line (remove the '#' at the beginning):

• Save and close the file.

3. Open the pg_hba.conf file:

• Add this line at the end of the file:

• Replace {host_IP} with the actual IP address of the machine running Apache Airflow.

• Save and close the file.

Restart PostgreSQL to apply changes:

MySQL Database Setup for Airflow

To configure MySQL as the database backend for Apache Airflow, follow these steps:

Install MySQL Server:

Install the mysqlclient Python package:

Start the MySQL service:

Install MySQL Connector for Python:

Secure MySQL Installation (Optional but Recommended):

Follow the on-screen prompts to complete the security setup.

Create a database and user for Airflow:

1. Access the MySQL shell:

Enter the root password when prompted.

2. Inside the MySQL shell, execute the following commands:

• Create the Airflow database:

• Create a user for Airflow and set permissions:

Restart MySQL to apply changes:

With these steps, the MySQL database named 'airflow' and a user named 'airflow' are now set up with the necessary privileges. You can now proceed to configure Apache Airflow to use this MySQL database as its backend.

Ubuntu 20/22 Setup

Prerequisites

Update the package list:

Add the Deadsnakes PPA repository to install newer Python versions:

Install Python 3.11 and the Python virtual environment package:

Verify the installation of Python and pip:

1. Check the Python version:

2. Check the pip version:

Database Setup

Follow the respective instructions below to configure your chosen database system for use with Apache Airflow.

PostgreSQL Database Setup

Install the psycopg2-binary Python package:

Install PostgreSQL:

Create a PostgreSQL database and user for Airflow:

1. Access the PostgreSQL shell:

2. Inside the PostgreSQL shell, execute the following commands:

• Create the Airflow database:

• Create the Airflow user with a password:

• Configure settings for the Airflow user:

• Grant all privileges on the Airflow database to the Airflow user:

• Exit the PostgreSQL shell:

Configure PostgreSQL settings for Airflow:

1. Open and edit the PostgreSQL configuration file:

• Change and uncomment the listen_addresses to allow all connections:

• Uncomment the port line to use the default port 5432:

2. Modify the pg_hba.conf file to allow specific connections:

• Add this line to permit connections from the Airflow server:

• Replace {host_IP} with the actual IP address of the machine running Apache Airflow.

Restart PostgreSQL to apply changes:

These steps have prepared your PostgreSQL database named 'airflow' and a user named 'airflow' with the necessary settings and privileges. You can now proceed to integrate this setup into Apache Airflow's configuration.

MySQL Database Setup for Airflow

Install MySQL Server:

1. Download and install the MySQL APT repository:

2. Update the package list and import the repository key:

3. Check the MySQL server version available and install it:

4. Install the MySQL Connector for Java:

Install the mysqlclient Python package:

Start the MySQL service:

Install MySQL Connector for Python:

Secure MySQL Installation (Optional but Recommended):

Run the command to secure your MySQL installation, including setting a root password:

Follow the on-screen prompts to complete the security setup.

Create a database and user for Airflow:

1. Access the MySQL shell:

Enter the root password when prompted.

2. Inside the MySQL shell, execute the following commands:

• Create the Airflow database with UTF-8 encoding:

• Create a user for Airflow and grant privileges:

Restart MySQL to apply changes:

With these steps completed, the MySQL database named 'airflow' and a user named 'airflow' are set up with the necessary privileges. You can now proceed to configure Apache Airflow to use this MySQL database as its backend.

Apache Airflow Installation using Mpack on Ambari

Create symbolic links for Python to use Python 3.11:

This following provides the steps for installing and setting up Apache Airflow using Management Pack (Mpack) on an Ambari-managed cluster.

Install and Configure Mpack:

1. Download the Airflow mpack from the Mpacks Link.

2. Run the following command to install Mpack:

2. Uninstall Previous Mpack (if needed):

3. Change Symlinks:

• Navigate to the services directory and update the Airflow symlink for each service version:

4. Restart Ambari Server:

Your Apache Airflow installation is now configured and ready for use on your Ambari-managed cluster.

Steps to install Apache Airflow from the Ambari UI

1. Add the Airflow service from the Ambari UI.

2. Specify the host details for the Airflow Scheduler and Airflow Webserver.

3. Choose the slave client configuration.

4. Modify or customize the fields as needed.

Database Options:

Choose between MySQL or PostgreSQL as the backend database:

Configuring the Airflow backend database connection string and Celery settings. Users will be prompted to input specific information, including the database name, password, username, database type (choose between MySQL or PostgreSQL), and host IP. The provided script will then automatically generate the necessary configuration details for the database connection string and Celery settings.

Enter Database Information in ambari UI.

• Database Name

• Password

• Username

• Database Type: Choose between mysql or postgresql.

• Host IP

If you are using RabbitMQ then you have to setup and add RabbitMQ configurations.

• RabbitMQ Username

• RabbitMQ Password

• RabbitMQ virtual host

• Celery Broker

Once you have provided all the necessary details, click on the Next button.

5. Deploy the Airflow service.

6. This step will install all the necessary components and initiate the service.

7. The Airflow webserver is up and running. To access the UI, you need to create a username and password. To create the admin user, you must run the initdb command from the Ambari UI.

On completion of the database initialization, you can access the Airflow Webserver UI. Enter the provided credentials ("airflow" as the username and "airflow" as the password) to log in and access the Airflow webserver UI.