3.13 Schema enforcement and Schema Evolution

   • Microsoft Fabric For Beginners  
In this tutorial, I explain how to handle schema irregularities in Spark when ingesting into Fabric Lakehouse tables. I also explain how to implement schema evolution for Lakehouse tables.

Chapters:
00:00- Introduction
01:48- Preview
03:53- Preparing data
05:34- Demo 1: Default schema enforcement in Spark
07:14- Disabling schema enforcement
07:59- Demo 2: Handling corrupt values
10:05- Demo 3: Exploring Permissive mode
10:47- Demo 4: Exploring Dropmalformed mode
11:17- Demo 5: Exploring Failfast mode
11:42- Demo 6: Handling schema mismatches on Lakehouse write
13:20- Demo: Using mergeSchema option
14:15- Demo: Using schema.autoMerge configuration setting

You can download the demo artifacts here:
Notebook: https://github.com/fazizov/youtube/bl...
Data: https://github.com/fazizov/youtube/bl...

Please subscribe:    / @fazizov  
Follow me on Linkedin: www.linkedin.com/in/fikrat-azizov-865a0528
and X: @fikrat_azizov

Official Documentation:
https://learn.microsoft.com/en-us/fab...


Hashtags:
#realtimeanalytics, #streaming, #microsoftfabric ,#azure,#microsoft, #dataengineering,#cloudcomputing, #dataanalytics, #lakehouse, #azuretutorial, #azuretraining, #datapipeline, #dataextraction , #dataintegration, #datatransfer, #dataflow, #spark, #deltalake, #synapse, #synapsedataenginering, #demo, #datalake, #transformation, #ingested, #datawarehouse, #dataintegration, #azuredatabricks ,#databricks, #bigdata, #bigdatatechnologies, #pyspark, #sparksql, #notebook ,#transformationvideo, #bronze, #medallion , #modeling, #facts, #silver, #gold, #historical data, #incremental