Skip to content
DP-700 Fabric Engineer Hub

Ingest Data with Dataflows Gen2 in Microsoft Fabric

Ingest Data with Dataflows Gen2 in Microsoft Fabric

Source

Summary

This module covers low-code/no-code ETL data extraction, visual wrangling, and ingestion using Dataflows Gen2 with Power Query Online in Microsoft Fabric, including fast copy optimization and data destination configurations.

Core Capabilities & Architecture

1. Dataflow Gen2 vs Dataflow Gen1

  • Data Destinations: Dataflows Gen2 can output transformed data directly into Fabric Lakehouses, Fabric Data Warehouses, Azure SQL Databases, and Azure Synapse Analytics.
  • Fast Copy Integration: Automatically leverages distributed backend compute to accelerate high-volume data transfers without choking on single-node mashup engine limits.
  • Staging Lakehouse: Uses an internal staging lakehouse and compute cluster to perform heavy transformations and query folding off-engine.

2. Output Destination Configuration

  • Update Mode:
    • Replace: Drops existing destination table and recreates with fresh schema and data.
    • Append: Appends new records into the destination table without schema modification.

3. Incremental Refresh Configuration

  • Configures partition filters based on Date/Time parameters (RangeStart and RangeEnd) to refresh only recently changed partitions rather than reloading entire datasets.

Exam Traps & Gotchas

[!WARNING]

  • Query Folding Importance: For optimal refresh performance, verify query folding indicators in the Power Query steps. Non-foldable operations (such as certain custom complex R/Python scripts or non-standard index operations) force row-by-row memory processing.
  • Destination Schema Drift: Choosing Append mode requires destination table schema alignment. If upstream schema introduces new columns, the pipeline will fail unless schema evolution is handled.