Скачать или смотреть Multi-Token Prediction (forget next token LLM?)

Meta published a new method of multi-token prediction for autoregressive transformer models (LLMs).

Additional heads perform in parallel token predictions. Benchmark data investigated and a special session for my green grasshoppers!

Instead of sequentially predicting the next token based on previously observed tokens, this architecture employs multiple output heads that operate in parallel from a shared trunk—the main body of the model which processes the input and generates a common latent representation. Each output head predicts a different future token independently, thereby enriching the model’s capability to parallel-process information and predict sequences more efficiently.

By predicting several tokens simultaneously, the model can better learn contextual dependencies between tokens in the sequence, resulting in more coherent and contextually accurate outputs. Additionally, this approach can enhance model robustness by diversifying the training objectives across multiple heads, thereby improving the model's generalization capabilities across different types of data and tasks.

All rights w/ authors of this arXiv pre-print:
https://arxiv.org/pdf/2404.19737

#airesearch