Automating Data Pipelines with Python & GitHub Actions [Code Walkthrough]

👉 More on Full Stack Data Science:    • Full Stack Data Science  

This is the 6th video in a series on Full Stack Data Science. Here, I use Python and GitHub actions to automate a data pipeline for FREE!

More Resources:
💻 Example Code: https://github.com/ShawhinT/data-pipe...
📰 Read more: https://towardsdatascience.com/automa...

🛠️ Data Engineering:    • Text Embeddings, Classification, and ...  
👨🏻‍💻 ML app repo: https://github.com/ShawhinT/yt-search
🔍 ML app UI: https://huggingface.co/spaces/shawhin...

--
Homepage: https://www.shawhintalebi.com/

Intro - 0:00
Motivation - 0:32
2 Ways to Automate - 1:28
Way 1: Orchestration Tool - 2:00
Way 2: Python + Triggers - 3:38
GitHub Actions - 5:56
Example Code: Automating ETL Pipeline - 7:42
1) Create ETL Python Script - 8:33
2) Create GitHub Repo - 12:21
3) Create Workflow .yml File - 13:22
4) Add Repo Secrets - 23:50
5) Commit and Push - 25:59
Final ML App - 28:45