Use Apache Spark in Microsoft Fabric

Use Apache Spark in Microsoft Fabric

Source

• Provider: Microsoft
• Platform: Microsoft Learn
• Source title: Use Apache Spark to work with files in a lakehouse
• Source URL: https://learn.microsoft.com/en-us/training/modules/use-apache-spark-work-files-lakehouse/
• Verification Link: https://learn.microsoft.com/api/achievements/share/en-us/pcoletsos/D3XVXQBJ?sharingId=F7F61A3006E08628
• Content type: Self-Paced Training Module & Achievement Badge
• Target Exam: Exam DP-700
• Date completed: 2026-08-04
• Last reviewed: 2026-08-15

Summary

This module covers running interactive data analysis, DataFrame transformations, and distributed processing using Apache Spark / PySpark notebooks inside Microsoft Fabric Lakehouses.

Key Architectural & Coding Concepts

1. PySpark DataFrame Operations

# Reading raw files from OneLake Lakehouse Files/ directory
df = spark.read.format("csv") \
    .option("header", "true") \
    .option("inferSchema", "true") \
    .load("Files/raw/sales_data.csv")

# Filtering and Column Transformations
from pyspark.sql.functions import col, to_date, upper, year

transformed_df = df.filter(col("OrderTotal") > 0) \
    .withColumn("OrderDate", to_date(col("OrderDate"), "yyyy-MM-dd")) \
    .withColumn("CustomerCountry", upper(col("CustomerCountry"))) \
    .withColumn("OrderYear", year(col("OrderDate")))

# Saving as a managed Delta Table in OneLake
transformed_df.write.format("delta") \
    .mode("overwrite") \
    .partitionBy("OrderYear") \
    .saveAsTable("fact_sales")

2. Spark Compute Pools in Fabric

• Starter Pools: Pre-warmed, rapidly provisioning compute instances (typically startup $<10$ seconds) ideal for interactive development.
• Custom Spark Pools: Dedicated pool configurations where you specify node family, minimum/maximum nodes (auto-scaling), and auto-pause timeout.
• Environment Items: Centralized management of public (PyPI, Conda) and private Python/R packages and custom Spark configuration properties.

3. Spark SQL and Metastore Integration

• Tables saved via saveAsTable() are immediately recorded in the Lakehouse metastore and queryable via %sql magic commands or the SQL analytics endpoint.

Exam Traps & Best Practices

[!TIP]

• Partitioning Best Practice: Avoid over-partitioning tables. Only partition tables larger than 1 TB, or columns with high cardinality where individual partition files exceed 1 GB.
• Display vs Show: Use display(df) in Fabric notebooks for interactive, sortable, and chartable table views instead of standard Spark df.show().

Related

• Work with Delta Lake Tables in Microsoft Fabric
• Organize a Fabric Lakehouse using Medallion Architecture Design
• Exam DP-700
• _indexes/modules-index.md