Fine-tuning on Wikipedia Datasets

➡️ Get Life-time Access to the Complete Scripts (and future improvements): https://Trelis.com/ADVANCED-fine-tuning/
➡️ One-click fine-tuning and LLM templates: https://github.com/TrelisResearch/one...
➡️ Trelis Livestreams: Thursdays 5 pm Irish time on YouTube and X.
➡️ Newsletter: https://blog.Trelis.com
➡️ Resources/Support/Discord: https://Trelis.com/About

VIDEO RESOURCES:
- Slides: https://docs.google.com/presentation/...
- Dataset: https://huggingface.co/datasets/Treli...
- WikiExtractor: https://github.com/attardi/wikiextractor

TIMESTAMPS:
0:00 Fine-tuning Llama 3 for a low resource language
0:40 Overview of Wikipedia Dataset and Loss Curves
1:53 Video overview
3:07 HuggingFace Dataset creation with WikiExtractor
12:11 Llama 3 fine-tuning setup, incl. LoRA
24:38 Dataset blending to avoid catastrophic forgetting
28:16 Trainer setup and parameter selection
34:40 Inspection of losses and results
36:43 Learning Rates and Annealing
42:27 Further tips and improvements