Apache Airflow Mpack Installation (Python3.11 & RHEL8 with Postgres15)

To install Apache Airflow management pack (mpack) for Python3.11 and RHEL8 with Postgres15, perform the following steps:

1. Install Python3.11 and create symlinks:

sudo ln -sf /usr/bin/python3.11 /usr/bin/python3

sudo ln -sf /usr/bin/pip3.11 /usr/bin/pip3

2. Install and setup Postgres15:

sudo -u postgres psql

CREATE DATABASE airflow;

CREATE USER airflow WITH PASSWORD 'airflow';

ALTER ROLE airflow SET client_encoding TO 'utf8';

ALTER ROLE airflow SET default_transaction_isolation TO 'read committed';

ALTER ROLE airflow SET timezone TO 'UTC';

GRANT ALL PRIVILEGES ON DATABASE airflow TO airflow;

\c airflow

GRANT ALL ON SCHEMA public TO airflow;

\q

3. Update PostgreSQL configuration files:

• Edit /var/lib/pgsql/15/data/postgresql.conf:

vi /var/lib/pgsql/15/data/postgresql.conf

• Uncomment the following lines:

listen_addresses = '*'

port = 5432

• Edit /var/lib/pgsql/15/data/pg_hba.conf:

vi /var/lib/pgsql/15/data/pg_hba.conf

• Add the following line:

host    airflow         airflow         37.27.118.52/32           md5

4. Restart PostgreSQL:

sudo systemctl restart postgresql-15

5. Install google-re2 package and psycopg2-binary:

pip3 install psycopg2-binary

pip3 install google-re2

If the package is not installed, download it and then install it. After running the below command, you will get a wheel package like google_re2-1.1.20240501-1-cp39-cp39-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl:

pip3 download google-re2

• Install Mpack:

ambari-server install-mpack --mpack=ambari-mpacks-airflow-2.8.1.tar.gz --verbose

• Refer to the screenshots below for guidance on installing the service via Ambari UI:

• Follow the Airflow configurations steps provided in the Apache Airflow Installation using Mpack on Ambari guide.

• Access the Airflow UI using your Airflow credentials, in this case we are using 'airflow' as the username and password.