Airflow - XCom

In Airflow, XCom stands for “Cross-Communication”. It’s a mechanism that allows tasks to exchange messages or small pieces of data.

---

Key Points about XCom

1. Purpose: Share data between tasks in a DAG.

2. Stored in: Airflow’s metadata database.

3. Size limitation: Usually small payloads (not meant for large files — use GCS/S3 for big data).

---

How it works

• A task can push a value to XCom:

ti.xcom_push(key='result', value=42)

• Another task can pull that value:

value = ti.xcom_pull(key='result', task_ids='previous_task')
print(value)  # 42

• ti is short for TaskInstance, which represents the currently running task.

---

Common Use Cases

• Passing results from one task to another.

• Passing parameters dynamically at runtime.

• Storing metadata, like IDs or URLs of generated resources.

---

Analogy

Think of XCom like a small bulletin board where tasks post messages for each other to read.