Скачать или смотреть Build the Llama Tokenizer from scratch

In this video, I'll walk you through the process of implementing the tokenizer used in the LLaMa language model. You'll learn the fundamentals of tokenization and build your own tokenizer step-by-step.
By the end, you'll have a solid understanding of how tokenizers work under the hood and the skills to build your own from scratch.
If you're interested in natural language processing or just want to level up your Python programming skills, check out this video.

0:00 - Introduction
3:05 - BPE Algorithm
4:28 - Video Structure
5:14 - Task 1
7:44 - Breakdown 1
8:15 - Breakdown 2
8:58 - Breakdown 3
10:11 - Breakdown 4
10:18 - Task 2
10:54 - Breakdown 1
11:52 - Breakdown 2
12:43 - Task 3
12:59 - Breakdown 1
13:09 - Breakdown 2
14:28 - Task 4
15:12 - Breakdown 1
15:32 - Breakdown 2
18:14 - Task 5
19:24 - Breakdown 1
22:00 - Modularize the code
22:35 - Concept
24:06 - Task 6
24:11 - Breakdown 1
24:26 - Breakdown 2
24:38 - Breakdown 3
29:19 - Task 7
30:46 - Bonus