Скачать или смотреть How to build an automated data pipeline using Airflow, dbt, Postgres, and Superset (Windows 11 WSL)

How to build an automated ELT pipeline from ingesting live data from an API to creating a near-realtime dashboard, using Windows WSL Ubuntu 24 and Docker images of Apache Airflow 3.0 (the latest), dbt, Postgres, and Apache Superset. In this live coding tutorial, I walk you through the thinking process, and show you every step.

This tutorial gets all the tools communicating with each other but does not explore too much into the functionalities of each. This video is version 1. Improvements are underway.

Looking to get personalized mentorship for landing your next data role? Data Engineer Academy helped me get started in my career as a Data Engineer. Use my referral link to get 10% off of your tuition: https://my.dataengineeracademy.com/lo...

0:00 - Intro
1:25 - Installing Visual Studio Code
2:05 - Setting up a WSL Linux Instance
5:56 - Installing and Setting up Docker for WSL
7:02 - Weatherstack API
7:59 - API Request using Python
13:48 - Setting up Postgres using Docker Compose
17:48 - Ingest Data into Postgres with psycopg2
27:24 - Setting up Airflow using Docker Compose
34:19 - Creating a DAG for Data Ingestion
43:40 - Setting up dbt using Docker Compose
50:16 - Creating dbt Models for Analytics
57:26 - Creating a DAG for dbt
1:04:34 - Creating DAG dependencies
1:05:38 - Setting up Superset using Docker Compose
1:16:38 - Creating a Real-time Superset Dashboard