Скачать или смотреть 🎙️How to Run Big Models on Small GPUs | All about Quantization Podcast

This episode breaks down how to shrink giant LLMs so they actually run on real hardware. If you’ve ever trained a huge model only to realize it doesn’t fit on your GPU (or your customer’s GPU), this is your roadmap. We cover the why, when, and how of quantization plus the traps that waste days if you don’t know what to look for.

What's covered in the podcast
Why quantization matters for deployment (and why FP16 is a trap)
QAT vs PTQ: the only two paths, and which one you actually need
AWQ, GPTQ, GGUF: strengths, weaknesses, and where they fit
Calibration data: what “good” looks like and why it makes or breaks quality
Precision choices: W4A16 vs W8A16 vs W8A8
Real-world failures: corrupted weights, OOMs, and mismatched loaders
How to test a quantized model before it burns you in production

Key takeaways
Quantization is an engineering process, not magic because clean configs and good data matter
W4A16 (AWQ) is the default choice for most GPU deployments
GGUF is the move for Apple Silicon, CPUs, and edge devices
Validate outputs immediately after quantizing

The goal is a stable, fast, and good enough model vs the smallest model.

📖 Read the full post the podcast is based on...
👉 https://nyghtowl.substack.com/p/how-t...