Master Azure Databricks CI/CD in 2 Hours with Azure DevOps | Full End-to-End CI/CD Project in Azure

#cicd #azuredevops #devops #databricks #azuredatabricks


In this Video, Firstly, I have discussed about what is CI/CD, how we will be using it for deploying Azure Databricks notebook from dev to prod and the merging techniques that we are going to follow for building the CI/CD pipelines.

Then, I have discussed about the different environment setup required for building the CI/CD pipelines. Things like Integrating the Azure DevOps with Azure Databricks workspace, Branch protection and merging techniques are seen in this video with Demo. I have also discussed about the process of the Continuous Integration (CI) pipeline and created the setup required for creating the CI pipelines. Then, I have discussed about the process of the Continuous Integration (CI) pipeline and explained the different YAML code used for creating the pipeline and followed by the actual creation of the CI/CD pipeline using the YAMl file.

After creating the CI part, I have discussed about the Continuous Deployment part where it deploys changes from DEV to PROD databricks workspace

Finally, I have discussed about how to setup permissions for the PROD environment before deploying the changes in PROD and also the complete end to end CI/CD pipeline testing.

Chapters

TimeStamp:

0:00:00 - Intro
0:01:48 - What is CI/CD?
0:05:25 - Merging / Branching Techniques used in CI/CD
0:11:15 - Git Integration in Azure Databricks (Env Setup)
0:22:39 - Branch Protection (Env Setup)
0:40:01 - Continuous Integration (CI) Pipeline process explained
0:49:03 - Organizing the folder structure needed for CI / CD Pipeline
0:59:54 - YAML code for CI Pipeline and Variable Group, Pool-VM Image (Compute)
1:11:33 - Environments and Service connection
1:19:33 - Deploy Notebook Functionality and Generating Databricks Token
1:28:34 - Creating Pipeline in Azure DevOps
1:38:56 - Continuous Deployment (CD) Pipeline
1:56:37 - Prod Environment Protection and End to End CICD Pipeline Testing
2:04:58 - Outro

– – – Book a Private One on One Meeting with me (1 Hour) – – –

https://www.buymeacoffee.com/mrktalks...

– – – "Express your encouragement by brewing up a cup of support for me" – – –

https://www.buymeacoffee.com/mrktalks...


– – – Other useful playlist: – – –

1. Microsoft Fabric Playlist:    • Microsoft Fabric Tutorials  
2. Azure General Topics Playlist:    • Azure Beginner Tutorials  
3. Azure Data Factory Playlist:    • Azure Data Factory Tutorials  
4. Databricks CICD Playlist:    • CI/CD (Continuous Integration and Con...  
5. Azure Databricks Playlist:    • Azure Databricks Tutorials for Beginners  
6. Azure End to End Project Playlist:    • End to End Azure Data Engineering Rea...  
7. End to End Azure Data Engineering Project:    • An End to End Azure Data Engineering ...  

– – – Let’s Connect: – – –

Email: [email protected]
Instagram: mrk_talkstech


– – – Tools & Equipment (Gears I use): – – –

Disclaimer: Links included in this description might be affiliate links. If you purchase a product or service with the links that I provide, I may receive a small commission. There is no additional charge to you! Thank you for supporting me so I can continue to provide you with free content each week!

DJI Mic: https://amzn.to/3sNpDv8
Dell XPS 13 Plus 13.4" 3.5K : https://amzn.to/45KqH1c
Rode VideoMicro Vlogger Kit: https://amzn.to/3sVFW8Y
DJI Osmos Action 3: https://amzn.to/44KYV3x
DJI Mini 3 PRO: https://amzn.to/3PwRwAr


– – – About me: – – –

Mr. K is a passionate teacher created this channel for only one goal "TO HELP PEOPLE LEARN ABOUT THE MODERN DATA PLATFORM SOLUTIONS USING CLOUD TECHNOLOGIES"

I will be creating playlist which covers the below topics (with DEMO)

1. Azure Beginner Tutorials
2. Azure Data Factory
3. Azure Synapse Analytics
4. Azure Databricks
5. Microsoft Power BI
6. Azure Data Lake Gen2
7. Azure DevOps
8. GitHub (and several other topics)

After creating some basic foundational videos, I will be creating some of the videos with the real time scenarios / use case specific to the three common Data Fields,

1. Data Engineer
2. Data Analyst
3. Data Scientist

Can't wait to help people with my videos.

– – – Support me: – – –

Please Subscribe:    / @mr.ktalkstech  


#AzureDatabricks #ApacheSpark #Sparkcompute #clusters #notebooks #magiccommands #machinelearning #ETL