Installing Custom Packages in Apache Airflow

  1. Access the Node: Log into the node where the Apache Airflow Scheduler and Webserver are installed.

  2. Navigate to the Airflow Home Directory: Move to the directory designated as the Airflow home.

  3. Activate the Virtual Environment: Activate the virtual environment used by Airflow to ensure package installations are isolated and consistent.

source <virtual_environment_path>/bin/activate

  1. Install the Custom Packages: Use the appropriate package manager (e.g., pip) to install any required custom packages. For example:

pip install <package_name>
  1. Verify Installation: Ensure the installed packages are available by checking their version or importing them in a Python script.

  2. Deactivate the Virtual Environment: After installing the packages, deactivate the virtual environment by running:

deactivate
  1. Restart Airflow Services: Restart the Airflow services using the Ambari UI to apply the changes.


  Last updated