3.07 How to build and orchestrate Dynamic and Reusable Spark pipelines in Microsoft Fabric.

   • Microsoft Fabric For Beginners  
In this lesson I explain how to build dynamic and reusable Spark Pipelines in Microsoft Fabric. I explain parameterization options for Spark notebooks and demonstrate how to call notebooks from other notebooks.
I also demonstrate how to call parameterized Spark notebooks from Data Pipelines and build end-to- end orchestration.
Finally, I demonstrate usage of IPython widgets to make Spark notebooks interactive.
You can download the related demo notebook from here:
https://github.com/fazizov/youtube/bl...,
https://github.com/fazizov/youtube/bl...

Chapters:
00:00- Introduction
02:35- Preview
04:20- Demo- Adding parameter cells to notebooks
07:22- Calling parameterized notebooks from other notebooks
09-25- Orchestrating parameterized notebooks from Data Pipelines
14:48- Adding Interactivity experience to notebooks by using IPython widgets

Please subscribe:    / @fazizov  

Official Documentation:
https://learn.microsoft.com/en-us/fab...
https://learn.microsoft.com/en-us/fab...

Hashtags:
#datafactory, #microsoft,#microsoftfabric ,#azure, #dataengineering,#cloudcomputing, #dataanalytics, #lakehouse, #azuretutorial, #azuretraining, #datapipeline, #dataextraction , #dataintegration, #datatransfer, #dataflow, #spark, #deltalake, #synapse, #synapsedataenginering, #demo, #datalake, #transformation, #ingested, #datawarehouse, #dataintegration, #azuredatabricks ,#databricks, #bigdata, #bigdatatechnologies, #pyspark, #sparksql, #notebook ,#transformationvideo, #bronze, #medallion, #kimball, #dimensions , #modeling, #facts, #silver, #gold, #historical data, #dimensional