Python ETL pipeline how to perform incremental data load | Source Change Detection | Python | ETL

In this video we will cover how to perform Incremental data load with Python ETL pipeline. The incremental data load approach in ETL (Extract, Transform and Load) is the ideal design pattern. In this process, we identify and process new and modified rows since the last ETL run. Incremental data load is efficient in the sense that we only process a subset of rows and it utilizes less resources.
We will focus on Source Change Detection technique today.


Link to medium article on this topic: https://blog.devgenius.io/python-etl-...

Change Data Capture approach video: https://www.youtube.com/watch?v=FJSvK...

Insert on Conflict: https://www.postgresqltutorial.com/po...

Link to GitHub repo: https://github.com/hnawaz007/pythonda...

Videos in this series:
Build ETL pipeline:    • How to build an ETL pipeline with Pyt...  
Automate ETL Pipeline:    • How to build and automate your Python...  
ETL Load Reference Data:    • How to load reference data to databas...  
ETL Incremental Data Load (Destination Change Comparison):    • How to build ETL pipeline with Increm...  

SQL Database setup videos:
SQL Server setup:    • Install SQL Server Express 2019 Step ...  
PostgreSQL setup:    • How to install PostgreSQL  on windows...  

Subscribe to our channel:
   / haqnawaz  

---------------------------------------------
Follow me on social media!

Github: https://github.com/hnawaz007
Instagram:   / bi_insights.  .
LinkedIn:   / haq-nawaz  

---------------------------------------------

#ETL #Python #IncrementalDataLoad

Topics covered in this video:
0:00 - Introduction to ETL Incremental load approach
0:41 - Source Setup
1:04 - Target Setup
2:33 Implement Incremental load approach with Python
5:17 - Test Pipeline with target data check