OneLake Shortcuts: Virtual pointers referencing data in external storage (ADLS Gen2, Amazon S3, Google Cloud Storage) or internal Fabric items without moving or duplicating data.
Workspace vs Item: A workspace is an administrative, security, and Git-integration boundary; a Lakehouse, Warehouse, Eventhouse, or Semantic Model is an item inside a workspace.
Data Team Collaboration:
Data Engineer: Ingests and transforms data via pipelines, notebooks, and lakehouses.
Data Scientist: Explores data with Spark/Python and builds ML models.
Data Analyst: Builds semantic models and reports, leveraging Direct Lake.
Analytics Engineer: Curates Gold-layer data and enterprise semantic definitions.
2. Data Ingestion & Orchestration
Dataflow Gen2 vs Data Pipelines vs Copy Data
Feature
Copy Data Activity
Dataflow Gen2
Data Pipeline
Role
High-performance data mover
Low-code transformation worker
Orchestration manager
Engine
Data Factory cloud compute
Power Query Online mashup engine
Control-flow execution engine
Best For
Moving raw data largely as-is to staging/OneLake
Visual cleaning, reshaping, and merging for business users
Multi-step scheduling, loops, dependencies, retries, and failure alerts
Security / Storage
Does not store data
Lands data into LH/WH/SQL DB (not long-term storage)
Coordinates calls to Dataflows, Notebooks, Stored Procedures
[!TIP]
Worker vs. Manager Rule: Dataflow Gen2 and Spark Notebooks perform transformation work. A Data Pipeline coordinates, schedules, parameterizes, and connects workers.
Pipeline Parameters vs. Variables
Parameter: Passed from outside at runtime to make the pipeline reusable (e.g., passing Month = 'March' to process a specific partition).
Variable: Internal state holder evaluated or updated during a specific execution run (e.g., storing a row count or status flag).
3. Apache Spark & Notebooks
Architecture in Fabric
Head Node / Driver: Coordinates the application execution plan and distributes tasks.
Worker Nodes / Executors: Execute assigned tasks and process data in parallel.
Spark Pool: Managed compute cluster (starter pools offer rapid sub-10s spin up; custom pools allow specific VM sizing and auto-scale).
Environment: Custom library definition (PyPI/Conda/Wheels) and Spark configurations attached to notebooks/workspaces.
PySpark Operations & Spark SQL Equivalents
PySpark Operation
SQL Equivalent
Exam Context / Purpose
df.select("ColA", "ColB")
SELECT ColA, ColB
Column projection
df.where("Sales > 100") / df.filter(...)
WHERE Sales > 100
Row filtering
df.groupBy("Region").agg(...)
GROUP BY Region
Aggregation
df.write.mode("overwrite").saveAsTable(...)
Target table replacement
Full refresh
df.write.partitionBy("Year", "Month")
Folder-level physical layout
Partitioned storage
Spark SQL Catalog: Tables vs. Views
Local Temporary View (df.createOrReplaceTempView("view_name")): Session-scoped, ephemeral, not registered in the catalog.
Managed Table (df.write.saveAsTable("tbl")): Delta table registered in catalog; dropping table deletes metadata AND underlying files.
DENSE_RANK(): Equal values receive same rank; no numerical gaps.
NTILE(n): Splits partitioned dataset into $n$ equal ranked buckets.
APPROX_COUNT_DISTINCT(): High-performance approximate cardinality estimation for big data.
6. Real-Time Intelligence (RTI)
RTI Components
Real-Time Hub: Single catalog to discover, manage, and consume streaming data streams across the tenant.
Eventstream: No-code visual drag-and-drop tool to ingest, filter, transform (windowed aggregations, join, expand), and route events to Eventhouse, Lakehouse, or Activator.
Eventhouse / KQL Database: High-velocity columnar database optimized for append-heavy time-series telemetry and log analytics.
Activator: Rule and trigger engine monitoring live conditions across Power BI, Eventstreams, or KQL DBs to fire actions (Email, Teams, Power Automate, Pipeline trigger).
KQL Best Practices for DP-700
Filter Early: Place | where Timestamp >= ago(1h) as the first operator to leverage indexing and reduce partition scanning.
Project Narrowly: Apply | project-away or | project ColA, ColB early to minimize memory footprint.
Join Optimization: Always place the smaller dataset on the left of the | join operator (SmallTable | join kind=inner (LargeTable) on Key).
Materialized Views: Precompute heavy aggregations over append-heavy tables without full reprocessing.
Update Policy: Attach to destination curated table to process incoming batches in near-real-time.
7. Security, Roles & Access Control
Security Evaluation Order
Entra ID Authentication: Verifies identity and tenant access.
Fabric Workspace / Item Access: Controls whether the user can open the workspace or specific items.
Data Security: Controls which tables, rows, columns, or files the user is permitted to read.
Permission Hierarchy Rules
DENY Overrides GRANT: An explicit DENY takes precedence over group or role-level GRANT.
Item Sharing: Prefer targeted Item-level sharing (Read, ReadAll) over granting broad workspace roles to external viewers.
OneLake Security Roles: Apply unified RBAC across engines; users must be removed from DefaultReader for custom restricted roles to enforce restrictions.
SQL Security Snippets:
Dynamic Data Masking: ALTER COLUMN Email ADD MASKED WITH (FUNCTION = 'email()');
Row-Level Security: Inline table-valued predicate function bound via CREATE SECURITY POLICY.
Column-Level Security: DENY SELECT ON dbo.Table(SensitiveColumn) TO RoleName;
8. Monitoring & Performance
DMVs & Query Insights (Warehouse & SQL Endpoint)
Diagnostic Object
Purpose
sys.dm_exec_connections
Live active warehouse connections
sys.dm_exec_sessions
Authenticated user sessions
sys.dm_exec_requests
Active executing queries
queryinsights.exec_requests_history
Historical log of completed T-SQL queries
queryinsights.long_running_queries
Queries ranked by execution duration
queryinsights.frequently_run_queries
Queries ranked by execution frequency
queryinsights.exec_sessions_history
Historical session connection metadata
[!IMPORTANT]
To inspect other users’ active queries or terminate a runaway query with KILL <session_id>, Workspace Admin permissions are required.
9. CI/CD & Lifecycle Management
Three Pillars of Fabric Lifecycle
Git Integration: Synchronizes workspace items with GitHub or Azure DevOps branches for source control, peer review, and branching.
Deployment Pipelines: Promotes validated Fabric items between segregated environment workspaces (Development $\rightarrow$ Test $\rightarrow$ Production).
Fabric REST APIs & CLI: Programmatic automation of workspace synchronization and stage deployments.
Clean Deployment Architecture Pattern
Connect only the Development workspace directly to Git feature branches.
Use Pull Requests into the main branch for code review and approval.
Use Deployment Pipelines to promote validated artifacts from Development $\rightarrow$ Test $\rightarrow$ Production.
