Skip to content
DP-700 Fabric Engineer Hub

Monitor and Optimize Data Engineering Workloads in Fabric

Monitor and Optimize Data Engineering Workloads in Fabric

Source

Summary

This module covers diagnosing performance bottlenecks, monitoring distributed Spark jobs, analyzing query execution plans, and tuning Lakehouse and Warehouse workloads in Microsoft Fabric.

Monitoring & Optimization Tooling

1. Monitoring Hub & Spark Application UI

  • Fabric Monitoring Hub: Single pane of glass for real-time and historical status of pipeline runs, Spark notebook sessions, Dataflow refreshes, and ML experiments.
  • Spark Application UI & DAG Visualizer:
    • Inspects stages, jobs, and tasks.
    • Detects Data Skew (tasks where 1 executor takes significantly longer than the rest due to uneven partition keys).
    • Detects Spill to Disk (insufficient executor memory forcing Spark to write shuffle data to temporary disk).

2. High-Impact PySpark Optimization Techniques

# 1. Broadcast Join for small lookup dimension tables
from pyspark.sql.functions import broadcast


optimized_join_df = large_fact_df.join(
    broadcast(small_dim_df),
    on="CustomerKey",
    how="inner"
)


# 2. Adaptive Query Execution (AQE) configuration
spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "true")
spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true")


# 3. Delta File Compaction
spark.sql("OPTIMIZE fact_sales")

3. Fabric Warehouse Query Insights

  • queryinsights.exec_requests_history: Historical query execution metrics (duration, CPU time, rows affected).
  • queryinsights.frequently_run_queries: Identifies recurring high-consumption queries.
  • queryinsights.long_running_queries: Identifies queries running beyond normal SLAs.

Exam Traps & Gotchas

[!TIP]

  • Data Skew Resolution: When you see 99 tasks finish in 2 seconds and 1 task takes 10 minutes, apply salt keys (salting) or enable spark.sql.adaptive.skewJoin.enabled = true.
  • Broadcast Join Threshold: Spark defaults to 10MB threshold for automatic broadcasting (spark.sql.autoBroadcastJoinThreshold); explicitly use broadcast() for dimension tables up to ~100MB.