Скачать или смотреть Azure for DE: 37 - Orchestrating Databricks notebooks

Hey, data engineers! We have successfully developed Databricks notebooks for data transformation, and they are functioning smoothly. However, up to this point, we have been executing them manually, cell by cell. How can we automate this process and integrate it into our orchestration workflow?

Discover the solution by tuning in to the 37th episode of my free DP-203 series.

Enjoy!

▬▬▬▬▬▬ IMPORTANT LINKS ▬▬▬▬▬▬
My LinkedIn profile:   / piotr-tybulewicz-81a8793  
GitHub with my drawings: https://github.com/TybulOnAzure/DP-203
Databricks pricing: https://azure.microsoft.com/en-us/pri...
MS tutorial to run notebook from ADF: https://learn.microsoft.com/en-us/azu...
Compute permissions in Databricks: https://learn.microsoft.com/en-us/azu...

▬▬▬▬▬▬ MEMBERSHIP ▬▬▬▬▬▬
Join this channel to get access to perks:
   / @tybulonazure  

▬▬▬▬▬▬ CHAPTERS ▬▬▬▬▬▬
00:00 Introduction
00:28 Scenario
04:06 Sample notebook
05:59 Sample ADF pipeline
09:10 Cluster types
13:45 Authentication types
21:50 Using ADF managed identity
40:27 Using a job cluster
51:44 Parametrizing the notebook
56:53 Using Databricks jobs
1:01:18 Summary