Скачать или смотреть Inference-Time Hyper-Scaling with KV Cache Compression

Inference-Time Hyper-Scaling with KV Cache Compression

https://openreview.net/pdf?id=8ZiElzQxf1

This research introduces inference-time hyper-scaling, a technique that improves Large Language Model (LLM) reasoning by compressing the KV cache. Standard reasoning models are often bottlenecked by memory load and latency, but reducing the cache size allows them to generate more parallel or longer reasoning chains within the same compute budget. To achieve this, the authors developed Dynamic Memory Sparsification (DMS), a data-efficient method for retrofitting models to selectively evict less important tokens. Unlike training-free alternatives, DMS maintains high accuracy even at 8x compression ratios by utilizing a delayed eviction policy. Experiments across various Qwen-R1 and Llama models demonstrate significant performance gains on math, coding, and scientific benchmarks. Ultimately, the study shows that efficient attention mechanisms are essential for expanding the Pareto frontier of model intelligence during inference.

#ai #research #largelanguagemodels #inference #kvcache

Disclaimer: This video is generated with Google's NotebookLM.