Скачать или смотреть 20250122 Develop AWS Glue-Spark jobs locally with docker and VS Code

#AWS #Glue #ETL #DataEngineering #MachineLearning #MLOps
👉 Requirements: The Amazon Bin Image Dataset contains over 500,000 images, each with a metadata JSON file sized around 1-3KB. I plan to perform ETL (Extract, Transform, Load) and EDA (Exploratory Data Analysis) to support subsequent machine learning tasks.

👉 Prerequisites: In the previous demo video, I uploaded a subset of the metadata (10K out of 500K) to an S3 bucket. We used AWS Athena with Trino SQL (and the Amazon Ion Hive SerDe) to query and consolidate 17.9MB of 10,441 JSON files into 3.9MB of 21 SNAPPY-compressed Parquet files, optimizing for better Spark performance.
https://docs.google.com/document/d/1b...

👉 In this demo video, I developed AWS Glue jobs locally using the AWS Glue-Spark Docker image and VS Code on Windows OS to perform data analysis on the metadata. Why local Glue job development? It provides a cost-effective, flexible, and fast development process that mirrors the cloud environment, while offering robust debugging, customization, and tool integration.

To reproduce the result:
◾ Pull the Docker image in Docker Desktop
◾ Configure the VS Code workspace file
◾ Start a container and attach a VS Code window
◾ Run Glue-Spark scripts or notebooks

👉 Check the text format tutorial - https://docs.google.com/document/d/1F...
◾ Repo folder - https://github.com/nov05/udacity-nd00...
◾ VS Code workspace configuration file - https://github.com/nov05/udacity-nd00...
◾ Windows cmd to start the container - https://github.com/nov05/udacity-nd00...
◾ Test glue script - https://github.com/nov05/udacity-nd00...
◾ Amazon Bin Image Dataset metadata EDA notebook - https://github.com/nov05/udacity-nd00...