Speculative Decoding: When Two LLMs are Faster than One

Описание к видео Speculative Decoding: When Two LLMs are Faster than One

Try Voice Writer - speak your thoughts and let AI handle the grammar: https://voicewriter.io

Speculative decoding (or speculative sampling) is a new technique where a smaller LLM (the draft model) generates the easier tokens which are then verified by a larger one (the target model). This make the generation faster computation without sacrificing accuracy.

0:00 - Introduction
1:00 - Main Ideas
2:27 - Algorithm
4:48 - Rejection Sampling
7:52 - Why sample (q(x) - p(x))+
10:55 - Visualization and Results

Deepmind Paper: https://arxiv.org/abs/2302.01318
Google Paper: https://arxiv.org/abs/2211.17192

Комментарии

Информация по комментариям в разработке

Похожие видео