Skip to content
DP-700 Fabric Engineer Hub

Implement Data Pipelines with Apache Airflow in Fabric

Implement Data Pipelines with Apache Airflow in Fabric

Source

Summary

This module explores using native managed Apache Airflow Workspaces (Python-based DAG workflows) in Microsoft Fabric to schedule, orchestrate, and monitor complex multi-system data pipelines alongside native Fabric items.

Core Concepts & DAG Authoring

1. Fabric Apache Airflow Workspaces

  • Managed Airflow environment running directly within Microsoft Fabric tenant capacity.
  • Direct authentication integration with Microsoft Entra ID and Fabric item endpoints.

2. Fabric Airflow Operators

  • FabricRunItemOperator: Triggers notebooks, pipelines, spark jobs, or semantic models directly from Python DAGs.
  • OneLakeHook: Enables seamless file transfer and metadata inspection against OneLake ADLS Gen2 endpoints.
from datetime import datetime
from airflow import DAG
from fabric_airflow_plugin.operators.fabric import FabricRunItemOperator


with DAG(
    dag_id="fabric_daily_curation_pipeline",
    start_date=datetime(2026, 1, 1),
    schedule_interval="@daily",
    catchup=False,
) as dag:


    run_bronze_ingestion = FabricRunItemOperator(
        task_id="run_bronze_ingestion",
        workspace_id="a1b2c3d4-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
        item_id="e5f6g7h8-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
        item_type="DataPipeline",
    )


    run_silver_spark_notebook = FabricRunItemOperator(
        task_id="run_silver_spark_notebook",
        workspace_id="a1b2c3d4-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
        item_id="i9j0k1l2-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
        item_type="Notebook",
    )


    run_bronze_ingestion >> run_silver_spark_notebook

Exam Traps & Gotchas

[!TIP]

  • When to use Airflow vs Data Factory: Use Data Factory pipelines for graphical, low-code/no-code workflows and fast native copy connectors; use Apache Airflow when complex Python logic, dynamic DAG generation, or multi-cloud orchestration across external providers is required.