Airflow - Components

• MetaDatabase
• API Server
• Scheduler
• DAG-Processor
• Triggerer

 

✔ Airflow Scheduler

The Scheduler is the core component that orchestrates your workflows. It is responsible for:

What it does

• Evaluates DAGs: Determines what tasks need to be run.

• Creates task instances based on schedules or callbacks.

• Queues tasks for execution by workers (Celery/Kubernetes/Local).

• Monitors task states (success, failure, retries).

• Handles SLAs, task dependencies, backfills, and scheduling intervals.

Key point

The scheduler decides when tasks should run and sends them to the executor.

---

✔ Airflow Triggerer

The Triggerer is used only for deferrable operators, introduced to handle async triggers efficiently.

What it does

• Runs async triggers that wait for events (e.g., sensor completion, external conditions) without blocking worker slots.

• Uses asyncio to handle thousands of concurrent lightweight tasks.

• Wakes the scheduler/executor when an event is completed.

Key point

The Triggerer waits for external events asynchronously, freeing worker resources.

Examples of things the triggerer waits for:

• File sensor waiting for object in S3

• Time-based sleep

• External API response

• Custom deferrable operator

---

✔ Airflow DAG Processor

The DAG Processor handles parsing and processing DAG files.

What it does

• Watches your DAG directory for new/changed DAG files.

• Parses DAG Python files in isolation, producing serialized DAGs stored in the Metadata DB.

• Offloads DAG parsing from the Scheduler for performance and safety.

Key point

The DAG Processor parses and prepares DAGs, while the Scheduler uses the parsed DAGs to schedule tasks.

---

🎯 Summary Chart

Component	Main Purpose	Key Responsibilities
Scheduler	Decides when tasks run	Create task instances, queue tasks, enforce dependencies
Triggerer	Handles async waiting	Manage deferrable operators and event-based waiting
DAG Processor	Parses DAGs	Scan DAG folder, parse files, serialize DAGs

---

🧠 Simple Analogy

• DAG Processor = “The compiler” → reads DAG code and prepares it.

• Scheduler = “The planner” → decides which tasks to run and when.

• Triggerer = “The waiter” → waits for asynchronous events efficiently.