Getting started with Dagster | Create Python ETL | Orchestrate ETL Pipelines with Dagster

In this video, we will cover an exciting new application called Dagster. It used to orchestrate your Python pipelines. Dagster has a user-friendly user interface and gives us better options of logging and history of the jobs we run with it. Dagster comes as a python library and you can quickly get setup and running with it.

Get started with Dagster in just three quick steps: Install Dagster, Define Ops and Materialize the assets.

Create a virtual environment: python -m venv env
Activate the virtual environment: env\Scripts\activate

To install Dagster into an existing Python environment, run: pip install dagster dagit

For projects using newer version 1.1.20 or 0.17.20 the command to create a new project has changed. To get started, you can run:
pip install dagster
dagster project scaffold --name my-dagster-project

Additional libraries required: Pandas, psycopg2

Create a new project: dagster new-project etl
CLI commands to run Dagit and daemon (run these commands in the same folder where the workspace.yml file is located):
dagit
dagster-daemon run

Access Dagit UI on port 3000: http://127.0.0.1:3000

Link to code, GitHub: https://github.com/hnawaz007/pythonda...

Subscribe to our channel:
   / haqnawaz  

---------------------------------------------
Follow me on social media!

GitHub: https://github.com/hnawaz007
Instagram:   / bi_insights_inc  
LinkedIn:   / haq-nawaz  

---------------------------------------------

#Python #ETL #Dagster

Topics covered in this video:
0:00 - Introduction ETL with Dagster
1:17 - ETL Direct Acyclic Graph (DAG)
2:25 - Dagster Setup
3:32 - Dagster Project Overview
4:48 - Run Dagster
5:25 - Dagster UI Overview
6:47 - Write Python ETL Pipeline with Dagster
11:18 - Run ETL Pipeline from Dagster UI
12:55 - Run relatively Large dataset test