: Use the XComObjectStorageBackend to store larger data exclusively in S3 or GCS while only keeping a reference in the metadata DB.
@task(retries=0) def fetch_transactions(**context): df = query_db() # Push allowed only to key "raw_txns" context["ti"].xcom_push(key="raw_txns", value=df.to_json()) return "done"
This comprehensive guide covers everything you need to know about Apache Airflow XComs (Cross-Communications). It explores how they work under the hood, standard usage patterns, security considerations, and advanced custom backends. What is an Airflow XCom? airflow xcom exclusive
To implement this, you must write a custom Python class inheriting from BaseXCom :
When you pull without specifying a task_id , you get the latest XCom from any task. To enforce exclusivity, always specify the exact source task. : Use the XComObjectStorageBackend to store larger data
Introduced in Airflow 2.0, the TaskFlow API ( @task decorator) abstracts away explicit XCom calls entirely. It allows you to pass outputs as inputs directly, mimicking standard Python programming.
: The specific execution instance (DAG run) of the pipeline. The Explicit vs. Implicit Paradox XComs can be pushed and pulled in two ways: What is an Airflow XCom
# Task A and Task B run in parallel task_a >> task_c task_b >> task_c