Coding Llama 3 from scratch in PyTorch - Part 2

In this video series, you will learn how to train and fine-tune Llama 3 model from scratch.

The goal is to code LLaMA 3 from scratch in PyTorch to create models with sizes 3B, 6B, 22B, 45B, 35B and 45BM params. In this second video, you'll learn about continous pretraining, LLM benchmarks and you'll also get to see the results.

🤖 Models:
Llama-3-6B-v0.1: https://huggingface.co/prince-canuma/...
Llama-3-6B-v0.1 adapters: https://huggingface.co/prince-canuma/...
Llama-3-6B-v0 (Untrained): https://huggingface.co/prince-canuma/...

📚Papers:
LoRA: Low-Rank Adaptation of Large Language Models: https://arxiv.org/abs/2106.09685
QLoRA: Efficient Finetuning of Quantized LLMs
: https://arxiv.org/abs/2305.14314


💻 To follow along you can use this colab notebook:
https://github.com/Blaizzy/Coding-LLM...

🎥 Coding Llama 3 from scratch video series
Part 1:    • Coding Llama 3 from scratch in PyTorc...