Скачать или смотреть Northcoders Bootcamp - Final project Video

ETL Data Platform for ToteSys to AWS Data Warehouse

This project involved designing and implementing a data platform to extract, transform, and load (ETL) data from ToteSys (a PostgreSQL database) into a Data Warehouse hosted on AWS. The system runs on a 30-minute schedule and follows these steps:

Extraction: A Lambda function extracts data from ToteSys and stores it in the Ingestion S3 bucket.
Transformation: Another Lambda function processes the latest data from Ingestion S3, converts it into a star schema, and saves the transformed data in the Processed S3 bucket.
Loading: A final Lambda function loads the processed data from Processed S3 into the Data Warehouse.

Tech Stack & Key Components:
Python: v3.12
AWS Services:
S3: Storage for raw and transformed data.
Lambda: Serverless compute for ETL tasks.
CloudWatch: Monitoring and logging.
EventBridge: Triggers for scheduled execution.
Step Functions: Workflow orchestration.
IAM: Manages permissions for various services
Secrets Manager: Securely stores credentials
Pandas: Used for data transformation.
Data Formats: JSON (raw ingestion) and Parquet (processed storage).
Infrastructure as Code (IaC): Terraform for provisioning AWS resources.
CI/CD: Automated deployment and testing.

This system ensures efficient and reliable data movement from ToteSys to the Data Warehouse, supporting analytics and reporting needs.