Building a Batch Data Pipeline using Airflow, Spark, EMR & Snowflake

In this project we will demonstrate the use of:
✅Airflow to orchestrate and manage the data pipeline
✅AWS EMR for the heavy data processing
✅Use Airflow to create the EMR cluster, and then terminate once the processing is complete to save on cost.

Prerequisite:
---------------------
Transient Cluster on AWS from Scratch using boto3 | Trigger Spark job from AWS Lambda
   • Transient Cluster on AWS from Scratch...  
Connect Apache Airflow to Snowflake Data Warehouse
   • Connect Apache Airflow to Snowflake D...  
Reference Architecture for Batch ETL Workloads using AWS EMR & Best Practices
  / reference-architecture-for-batch-etl-workl...  


Code:
---------------
https://github.com/SatadruMukherjee/D...
https://github.com/SatadruMukherjee/D...
https://github.com/SatadruMukherjee/D...
https://github.com/SatadruMukherjee/D...


Check this playlist for more Data Engineering related videos:
   • Demystifying Data Engineering with Cl...  

Apache Kafka form scratch
   • Apache Kafka for Python Developers  

Snowflake Complete Course from scratch with End-to-End Project with in-depth explanation--
https://doc.clickup.com/37466271/d/h/...

🙏🙏🙏🙏🙏🙏🙏🙏
YOU JUST NEED TO DO
3 THINGS to support my channel
LIKE
SHARE
&
SUBSCRIBE
TO MY YOUTUBE CHANNEL