Apache Airflow Logging Guide

Overview

Apache Airflow 2.8.1 generates logs for various components such as the Scheduler, Webserver, and Worker processes. These logs are essential for debugging and monitoring the system.

Log Storage Location

The Airflow logs are stored under the following directory:

xxxxxxxxxx

_var_log_airflow_logs_

To list the logs available, use:

xxxxxxxxxx

ls _var_log_airflow_logs_

This directory contains logs for various DAGs, the scheduler, and the webserver.

Scheduler Logs

The Airflow scheduler logs can be found in:

xxxxxxxxxx

_var_log_airflow_logs_scheduler_

The logs are organized by date, with the latest logs available under:

xxxxxxxxxx

_var_log_airflow_logs_scheduler_latest_

To inspect a specific log file, navigate to the corresponding date:

xxxxxxxxxx

ls _var_log_airflow_logs_scheduler_2025-02-19

Example output:

xxxxxxxxxx

clean_up.py.log

Webserver Logs

The Airflow webserver logs can be found in:

xxxxxxxxxx

_var_log_airflow_logs_webserver-access.log

_var_log_airflow_logs_webserver-error.log

To check the last 100 lines of the webserver logs, use:

xxxxxxxxxx

tail -n 100 _var_log_airflow_logs_webserver-access.log

For errors, check:

xxxxxxxxxx

tail -n 100 _var_log_airflow_logs_webserver-error.log

DAG Logs

Each DAG execution logs its runs under:

xxxxxxxxxx

_var_log_airflow_logs_dag_id=<dag_id>_run_id=<run_id>_task_id=<task_id>_attempt=<attempt>.log

Example:

xxxxxxxxxx

ls _var_log_airflow_logs_dag_id=dataset_consumes_1_run_id=manual__2025-02-14T08:04:06.983781+00:00_task_id=consuming_1_

Output:

xxxxxxxxxx

attempt=1.log

To view the logs of a task attempt:

cat _var_log_airflow_logs_dag_id=dataset_consumes_1_run_id=manual__2025-02-14T08:04:06.983781+00:00_task_id=consuming_1_attempt=1.log

Debugging with Systemd

Since Airflow components are managed using systemd, you can check the status and logs using:

Webserver

Check the webserver status:

xxxxxxxxxx

systemctl status airflow-webserver

View the last 100 logs:

xxxxxxxxxx

journalctl -u airflow-webserver -n 100 --no-pager

Scheduler

Check the scheduler status:

xxxxxxxxxx

systemctl status airflow-webserver

View the last 100 logs:

xxxxxxxxxx

journalctl -u airflow-webserver -n 100 --no-pager

Worker (if applicable)

If using Celery workers, check their status:

xxxxxxxxxx

systemctl status airflow-worker

View the last 100 logs:

xxxxxxxxxx

journalctl -u airflow-worker -n 100 --no-pager