Скачать или смотреть Incremental Loading in Apache Airflow - Handle Large Data More Efficiently

Big data can be difficult data. Design your Airflow pipelines to process data more efficiently with incremental loading.

We'll see how to use template variables to process small chunks of our source data.

👉 Sign up for the newsletter: https://kpdata.dev/newsletter/

Chapters:
00:00 Introduction
00:25 What is Incremental Loading?
01:53 Airflow Data Interval
03:50 Example DAG: Donut App Orders
07:05 Building the DAG
10:18 Rendering Template Variables
11:49 Re-Running a DAG
14:53 Outro

Code:
https://github.com/kishanpatel789/kp_...

Blog post:
https://kpdata.dev/blog/airflow-incre...

Other Resources
Airflow Docs: https://airflow.apache.org/docs/apach...
Airflow Template Variables: https://airflow.apache.org/docs/apach...

#airflow #dataengineering #python