Textbooks Are All You Need

Phi-1 is a new, smaller language model for code with 1.3B parameters, trained on a selection of web data and synthetically generated textbooks and exercises, achieving high accuracy on HumanEval and MBPP.

00:00 Section: 1 Introduction
03:48 Section: More related works
07:29 Section: 2.1 Filtering of existing code datasets using a transformer-based classifier
09:53 Section: 2.2 Creation of synthetic textbook-quality datasets
12:01 Section: 2.3 Model architecture and training
14:46 Section: Pretraining.
17:12 Section: 3 Spikes of model capability after finetuning on CodeExercises
20:06 Section: 3.2 Finetuning improves the model’s ability to use external libraries
22:22 Section: TKinter example.
24:08 Section: Chat mode example.
27:46 Section: 5 Data pruning for unbiased performance evaluation
28:48 Section: 5.1 N-gram overlap

https://arxiv.org/abs//2306.11644

YouTube:    / @arxivpapers  

PODCASTS:
Apple Podcasts: https://podcasts.apple.com/us/podcast...
Spotify: https://podcasters.spotify.com/pod/sh...