Скачать или смотреть Day 9 | 24 Days of Microsoft Fabric – Build SCD Type 2 Incremental Load Using Notebook

Welcome to Day 9 of the 24 Days of Fabric series! 🚀
In this session, I’ll show you how to implement SCD Type 2 (Slowly Changing Dimension) logic using Fabric Notebooks (PySpark) and Delta Tables in the Lakehouse — step by step.


---

🔍 What You’ll Learn

✅ Understand the SCD Type 2 concept — how to track history when data changes
✅ Load and transform data in Lakehouse using Notebooks
✅ Compare source vs target and detect changes automatically
✅ Use PySpark + Delta MERGE to update, insert, and close records
✅ End with a fully functional incremental pipeline that preserves data history


---

🧩 Key Steps Covered

1️⃣ Setup Lakehouse and upload source file (CSV/Parquet)
2️⃣ Read source data using PySpark
3️⃣ Load existing Delta table as target
4️⃣ Detect new and changed records
5️⃣ Apply SCD2 merge logic with IsCurrent, StartDate, and EndDate
6️⃣ Append new rows and close old ones
7️⃣ Verify historical version tracking


---

🧰 Tools Used

Microsoft Fabric Notebooks

Lakehouse (Delta Tables)

PySpark + DeltaTable

SQL-based incremental logic



---

📢 Next Up (Day 10)

👉 Dynamic Notebook & Pipeline Parameterization in Fabric — building reusable dataflows with config-driven logic.


---

#MicrosoftFabric #Notebook #Lakehouse #SCD2 #IncrementalLoad #DeltaTables #PySpark #DataEngineering #DataPipeline #AzureFabric #MicrosoftDataPlatform #24DaysofFabric #ETL #LearnFabric