3.08 Introduction to Spark Structured Streaming in Microsoft Fabric

   • Microsoft Fabric For Beginners  
Are you curious on how to build real-time, low-latency streaming pipelines using Spark Structured Streaming in in Microsoft Fabric?
In this tutorial I explain core concepts of Spark Structured Streaming-internals, data sources and destinations and write modes.
I demonstrate building streaming pipelines to fetch data from source files and Azure Event Hubs into Fabric Lakehouse tables
I also demonstrate stream management and monitoring commands and features.

You can download the related demo notebook from here:
https://github.com/fazizov/youtube/bl...

Chapters:
00:00- Introduction
02:00- Preview
03:41- What is Spark Streaming
07:12- Delta Lake Lambda Architecture
08:01- How Spark Structured Streaming works?
09:43- Streaming source and destinations
11:16: Streaming write modes-append, complete, update
12:28- Demo- create data pipeline to fetch source files
16:12- Creating streaming pipeline to read from file sources
24:30- Managing streams- start, stop, measure progress
37:28- Demo 2-Creating streams to read from Event Hub
45:42- Monitoring streaming jobs from Spark UI

Please subscribe:    / @fazizov  
Follow me on Linkedin: www.linkedin.com/in/fikrat-azizov-865a0528
and X: @fikrat_azizov

Official Documentation:
https://learn.microsoft.com/en-us/fab...
https://learn.microsoft.com/en-us/fab...
https://spark.apache.org/docs/latest/...
Hashtags:
#datafactory, #microsoft,#microsoftfabric ,#azure, #dataengineering,#cloudcomputing, #dataanalytics, #lakehouse, #azuretutorial, #azuretraining, #datapipeline, #dataextraction , #dataintegration, #datatransfer, #dataflow, #spark, #deltalake, #synapse, #synapsedataenginering, #demo, #datalake, #transformation, #ingested, #datawarehouse, #dataintegration, #azuredatabricks ,#databricks, #bigdata, #bigdatatechnologies, #pyspark, #sparksql, #notebook ,#transformationvideo, #bronze, #medallion, #modeling, #facts, #silver, #gold, #historical data, #streaming