Скачать или смотреть I Got Sick and Stuck at Home this Weekend… So I Built GPT-2 Decoder Only TransformerFrom Scratch

Google Colab: https://colab.research.google.com/dri...

GitHub Project: https://github.com/martinyanev94/mask...


In this video, I build a GPT-2–style causal language model completely from scratch using pure PyTorch. I don’t rely on transformers, accelerate, or any high-level abstractions. Instead, I code each part step by step and explain how a modern autoregressive transformer actually works under the hood.

We start from the fundamentals and gradually assemble the full model. I show how token and positional embeddings are combined, how causal self-attention is implemented with proper masking, and how multi-head attention, feed-forward layers with GELU, residual connections, and LayerNorm come together to form a GPT-style architecture. Every design choice is motivated so you can see why the model behaves the way it does.

Once the architecture is in place, I train the model on a real text dataset using next-token prediction. I walk through the training process, discuss stability details like gradient clipping and AdamW, and then demonstrate autoregressive text generation, including temperature and top-k sampling. Finally, I turn the trained model into a simple interactive chat so we can talk to it and see what it learned.

Business contact: [email protected]