Skip to content
DP-700 Fabric Engineer Hub

Use Data Factory Pipelines in Microsoft Fabric

Use Data Factory Pipelines in Microsoft Fabric

Source

Summary

This module covers orchestrating complex end-to-end data workflows, ETL/ELT pipelines, and data movement at scale using Fabric Data Factory Pipelines.

Key Pipeline Activities & Control Flow

1. Data Movement & Ingestion

  • Copy Activity: High-throughput distributed data movement from 100+ connectors (on-premises, cloud, SaaS) to OneLake. Supports binary copy, columnar mapping, and partition-level parallelism.
  • Change Data Capture (CDC): Ingests real-time incremental change streams from transactional databases without full table scanning.

2. Control Flow & Orchestration

  • ForEach Activity: Iterates over an array of tables, files, or partitions in parallel (batch count 1–50) or sequentially.
  • Lookup & Get Metadata Activity: Queries database tables or inspects file directory metadata (child items, existence, size, last modified).
  • If Condition & Switch Activities: Conditional branching based on execution status, day of week, or data flags.
  • Notebook Activity: Triggers parameterized PySpark notebook execution with inputs and reads output variables.
  • Dataflow Activity: Triggers Dataflow Gen2 refreshes within the pipeline execution graph.
  • Web & Teams/Outlook Notification Activity: Invokes external REST APIs or sends Teams webhook alerts on pipeline failure.

3. Pipeline Parameters & Variables

  • System Variables: @pipeline().RunId, @pipeline().TriggerTime, @pipeline().DataFactory.
  • Dynamic Content Expressions: String interpolation, date math (@formatDateTime(utcnow(), 'yyyy-MM-dd')), and JSON parsing (@activity('LookupConfig').output.firstRow.BatchId).

Exam Traps & Gotchas

[!WARNING]

  • ForEach Inside ForEach: Data Factory pipelines do NOT support nesting a ForEach activity directly inside another ForEach activity. To achieve nested iteration, invoke an Execute Pipeline activity inside the outer loop.
  • Parallelism Throttling: Setting high batch counts on ForEach when querying rate-limited APIs or small SQL instances can cause timeout failures; configure sequential execution or reduce concurrency.