How to Automate Event-based End-to-End ETL Pipeline using AWS Glue & AWS Lambda | Data Engineering

#dataengineering #aws #automation #etl

Learn how to build automated End-to-End event-based ETL pipelines using AWS technologies.

In this demo,
1. we will build AWS S3 triggers for PUT action. This trigger will invoke the lambda function when a new object is placed in the S3 bucket.
2. We will set up an AWS lambda function that listens to S3 events and calls Glue job run with run-time parameters.
3. We will develop an AWS glue job with complete Extract, transform, and load logic.
4. finally, we will convert the visual ETL glue job into the script to refactor the input argument and code.

00:00 - Introduction.
02:10 - IAM Roles Creation.
04:04 - AWS Lambda creation.
04:52 - S3 trigger generation.
07:18 - Testing S3 trigger.
08:35 - Create Glue Job.
09:00 - ETL Flow creation and testing.
14:45 - AWS Lambda code to connect and trigger glue job.
17:35 - Converting Glue Visual ETL to Script with parameters.
19:36 - Testing End-to-End Event-based flow.

Glue ETL Pipeline Demo:    • How to Build Data Pipeline to Perform...  

LET'S CONNECT!
📰 LinkedIn ➔   / jayachandra-sekhar-reddy  
🐦 Twitter ➔   / reddyjaysekhar​  
📖Medium ➔   / jay-reddy  
📲 Substack➔ https://databracket.substack.com
💁Fiverr ➔ https://www.fiverr.com/jayreddy9

#bigdata #bigdatatutorialforbeginners #python #awscloud #lambda #automation #dataanalytics #data #cloudstorage