Apache Airflow Logging Guide

Overview

Apache Airflow 2.8.1 generates logs for various components such as the Scheduler, Webserver, and Worker processes. These logs are essential for debugging and monitoring the system.

Log Storage Location

The Airflow logs are stored under the following directory:

To list the logs available, use:

This directory contains logs for various DAGs, the scheduler, and the webserver.

Scheduler Logs

The Airflow scheduler logs can be found in:

The logs are organized by date, with the latest logs available under:

To inspect a specific log file, navigate to the corresponding date:

Example output:

Webserver Logs

The Airflow webserver logs can be found in:

To check the last 100 lines of the webserver logs, use:

For errors, check:

DAG Logs

Each DAG execution logs its runs under:

Example:

Output:

To view the logs of a task attempt:

Debugging with Systemd

Since Airflow components are managed using systemd, you can check the status and logs using:

Webserver

Check the webserver status:

View the last 100 logs:

Scheduler

Check the scheduler status:

View the last 100 logs:

Worker (if applicable)

If using Celery workers, check their status:

View the last 100 logs: