Скачать или смотреть Data Lab Infra - Part 1: Architecture Design

In this video, we'll learn how to design a modern data stack, built for home labs, freelancing data experts, or general on-premise needs. I'll discuss the options and decisions that I made, provide perspective on how this infrastructure should be used, and explain how you can fork and adapt my code to easily move from on-premise to cloud, on AWS, GCP, or Azure.

I hope you enjoy this series! If have you have any questions or suggestions, please comment below, or join Discord so we can have chat—I love to talk about this stuff, so don't be shy!

📺 Watch the full Data Lab Infra series here ►    • Data Lab Infra  

► In part 1, we discuss and design the architecture for a modern data stack, built for home labs, freelancing data experts, or general on-premise needs.
► In part 2, we setup two Terraform projects, first to bootstrap our infrastructure, providing a basic foundation for future deployments, and secondly to configure state storage using an S3 object store, so that we can update our platform infrastructure from any host with access to Proxmox.
► In part 3, we continue with the second Terraform project, setting up our platform infrastructure with three Docker VMs and a GitLab VM, building the required GitLab QCOW2 image with Packer, and configuring it to use its assigned Docker VM to launch GitLab runners for CI/CD.
► In part 4, we create a Docker Compose stack, where we deploy and configure all of our shared services, describing the development and production workflows, and showing how to use GitLab to store variables and secrets that can be used during deployment via CI/CD.
► In part 5, we illustrate the deployment workflow for application-specific services, describing how to provision the required resources, like a database and its credentials, via CI/CD tasks called directly on the infrastructure repository.

🔗 Resources:
💻 GitHub repo: https://github.com/DataLabTechTV/data...
📝 Blog post: https://datalabtechtv.com/posts/data-...
📚 Proxmox: https://pve.proxmox.com/pve-docs/
📚 Docker: https://docs.docker.com/
📚 Terraform: https://developer.hashicorp.com/terra...

🔔 Subscribe for more deep dives data science, data engineering, and all things data!
☁️ Follow the channel on Bluesky: https://bsky.app/profile/datalabtecht...
🎧 Join our community's Discord:   / discord  
👽 Join our subreddit:   / datalabtechtv  
💻 Check out the channel's GitHub: https://github.com/DataLabTechTV
🌐 Visit our website for blog posts: https://datalabtechtv.com

#devops #datastack #homelab #proxmox #terraform #docker #gitlab

00:00:00 Intro
00:01:08 Architecture Overview
00:01:16 L1: Foundation
00:03:27 L2: Platform
00:06:10 L3: Services
00:09:05 L4: Applications
00:09:46 Terraform State Storage
00:10:24 No Kubernetes?
00:11:05 Understanding Decisions
00:11:21 OpenTofu vs Terraform
00:12:23 Podman vs Docker
00:13:28 Secrets Management
00:18:58 Docker Registry
00:21:27 Configuration Management
00:24:26 Core Services
00:24:30 Portainer
00:24:54 PostgreSQL
00:25:07 DuckLake
00:25:28 Apache Kafka
00:29:06 No Apache Flink?
00:29:52 MLflow
00:30:08 Ollama
00:31:48 From On-Premise to Cloud
00:32:14 Cloud VMs
00:34:04 Cloud Storage
00:35:35 Cloud Containers
00:36:44 Cloud Git
00:37:08 Cloud PostgreSQL
00:37:22 Cloud DuckLake
00:37:32 Cloud Apache Kafka
00:37:44 Cloud MLflow
00:38:00 Cloud LLMs
00:38:52 Outro