Apache Airflow Logging Guide

Overview

Apache Airflow 2.8.1 generates logs for various components such as the Scheduler, Webserver, and Worker processes. These logs are essential for debugging and monitoring the system.

Log Storage Location

The Airflow logs are stored under the following directory:

_var_log_airflow_logs_

To list the logs available, use:

ls _var_log_airflow_logs_

This directory contains logs for various DAGs, the scheduler, and the webserver.

Scheduler Logs

The Airflow scheduler logs can be found in:

_var_log_airflow_logs_scheduler_

The logs are organized by date, with the latest logs available under:

_var_log_airflow_logs_scheduler_latest_

To inspect a specific log file, navigate to the corresponding date:

ls _var_log_airflow_logs_scheduler_2025-02-19

Example output:

clean_up.py.log

Webserver Logs

The Airflow webserver logs can be found in:

_var_log_airflow_logs_webserver-access.log

_var_log_airflow_logs_webserver-error.log

To check the last 100 lines of the webserver logs, use:

tail -n 100 _var_log_airflow_logs_webserver-access.log

For errors, check:

tail -n 100 _var_log_airflow_logs_webserver-error.log

DAG Logs

Each DAG execution logs its runs under:

_var_log_airflow_logs_dag_id=<dag_id>_run_id=<run_id>_task_id=<task_id>_attempt=<attempt>.log

Example:

ls _var_log_airflow_logs_dag_id=dataset_consumes_1_run_id=manual__2025-02-14T08:04:06.983781+00:00_task_id=consuming_1_

Output:

attempt=1.log

To view the logs of a task attempt:

cat _var_log_airflow_logs_dag_id=dataset_consumes_1_run_id=manual__2025-02-14T08:04:06.983781+00:00_task_id=consuming_1_attempt=1.log

Debugging with Systemd

Since Airflow components are managed using systemd, you can check the status and logs using:

Webserver

Check the webserver status:

systemctl status airflow-webserver

View the last 100 logs:

journalctl -u airflow-webserver -n 100 --no-pager

Scheduler

Check the scheduler status:

systemctl status airflow-webserver

View the last 100 logs:

journalctl -u airflow-webserver -n 100 --no-pager

Worker (if applicable)

If using Celery workers, check their status:

systemctl status airflow-worker

View the last 100 logs:

journalctl -u airflow-worker -n 100 --no-pager