Airflow - Deferrable Task

 

    t0a = TimeDeltaSensor(
        task_id='wait_some_seconds_async',
        delta=datetime.timedelta(seconds=2),
        deferrable=True
    )

What's the purpose of 'deferrable=True'?

 

In Airflow, the parameter deferrable=True means the sensor will run in deferrable mode, which makes it much more efficient than a normal (non-deferrable) sensor.

Here’s what it really means:

---

✅ What “deferrable” means

A deferrable task is a task that:

• Does not occupy a worker slot while waiting

• Uses an async triggerer process instead of a worker

• Frees resources while it waits for its condition

This is a major improvement over the classic sensor behavior.

---

🧠 Why is this important?

Traditional sensors (even with reschedule mode) still require periodic checks.
But deferrable sensors can “pause themselves” and wake only when needed.

So instead of:

🕑 “Wake up every X seconds and check if ready”

They work like:

🔔 “Tell me when it’s ready — I’ll sleep until then without using a worker”

---

📌 How it works with TimeDeltaSensor

Your example:

t0a = TimeDeltaSensor(
    task_id='wait_some_seconds_async',
    delta=datetime.timedelta(seconds=2),
    deferrable=True
)

With deferrable=True:

• The task starts

• It registers a trigger that says:
  “Wake me after 2 seconds”

• It releases the worker immediately

• After 2 seconds, the triggerer wakes it

• The task completes

✔ No worker CPU usage
✔ No periodic poking
✔ Much more scalable

---

📊 Comparison: Normal sensor vs Deferrable sensor

Feature	Normal Sensor	Deferrable Sensor
Uses worker slot while waiting	Yes (poke mode), sometimes	No
Uses Triggerer instead	No	Yes
Scalable for many long waits	❌ Hard	✔ Excellent
Poke interval required	Yes	No
When runs	Repeatedly	Only twice (start & finish)

---

🎯 Why you should use deferrable=True when possible

• Saves worker resources

• Improves scheduler performance

• Allows you to run many more long-wait sensors

• Recommended by Airflow community for sensors waiting minutes/hours

Especially useful for:

• Time sensors (e.g., wait for a time)

• External system checks (file exists, API ready, job finished)

• Anything that "waits"

---

✔ Summary

deferrable=True turns a sensor into an async, resource-efficient task that sleeps without using a worker and wakes only when the trigger fires. It is the modern, recommended way to run sensors in Airflow 2.4+.