Airflow - TimeSensor

 

    t2 = TimeSensor(
        task_id='timeout_after_second_date_in_the_future',
        timeout=1,
        soft_fail=True,
        target_time=(datetime.datetime.now(tz=datetime.timezone.utc) + datetime.timedelta(hours=1)).time()
    )

 

In this Airflow task, the parameter timeout=1 refers to:

✅ The maximum amount of time (in seconds) that the sensor is allowed to run before Airflow marks it as failed or soft-failed.

---

⏱️ What exactly happens here?

• TimeSensor waits until the system time reaches target_time.

• But in your example, target_time is 1 hour in the future.

• However, timeout=1 means:

The sensor is allowed to wait only 1 second before it times out.

So:

• The sensor starts running.

• It checks the time.

• It sees that the target time is far in the future.

• After 1 second, the sensor times out because the target is still not reached.

---

📌 How Airflow handles this timeout

Because you also set:

soft_fail=True

Then:

• After 1 second, the sensor soft fails → task ends with status SKIPPED instead of FAILED.

If soft_fail=False, it would become FAILED.

---

💡 Why this is used?

This combination:

timeout=1
soft_fail=True

is a common pattern when:

• You want to skip a task or DAG branch depending on time conditions

• You want the task to not block workers waiting for long periods

• You want to quickly move on if the target time is too far away

Essentially, it becomes a time-based conditional skip.

---

📝 Summary

Parameter	Meaning
timeout=1	Sensor can only run for 1 second before giving up
soft_fail=True	On timeout, mark task as SKIPPED instead of FAILED
Result	The task will skip after 1 second because the target_time is too far in the future

 

    t4 = BashSensor(
        task_id='Sensor_fails_after_3_seconds',
        timeout=3,
        soft_fail=True,
        bash_command='exit 1'
    )

What will be the state of this task? FAILED or SKIPPED?

 

To determine whether the final state is FAILED or SKIPPED, we need to understand how Airflow handles:

• Sensor failures

• Sensor timeouts

• soft_fail=True

---

✅ Step-by-step reasoning

1️⃣ BashSensor pokes by running the Bash command

• Each poke runs:

  exit 1

• Exit code 1 = non-zero = failure

• For a BashSensor, a non-zero exit code means the poke returns False (condition not met).

So this sensor will never succeed.

---

2️⃣ timeout=3

• The BashSensor will keep poking until 3 seconds have passed.

• After 3 seconds, Airflow marks the sensor as timed out.

Timeouts are considered a type of failure.

---

3️⃣ soft_fail=True

Here is the key rule:

For sensors, when a timeout or a failure occurs, if soft_fail=True, the final state becomes SKIPPED, not FAILED.

This applies to both:

• poke failures

• timeouts

---

🎯 Final Answer

✅ The task will end in state: SKIPPED

Because:

• The bash command always exits with error → sensor never succeeds

• After 3 seconds, timeout occurs → sensor fails

• soft_fail=True converts this failure into SKIPPED

---

📝 Summary Table

Condition	Normal Behavior	With soft_fail=True
Bash command fails (exit != 0)	FAILED	SKIPPED
Sensor times out	FAILED	SKIPPED
Sensor condition is met	SUCCESS	SUCCESS

Your task hits a timeout, so with soft fail → SKIPPED.