Скачать или смотреть Kimi Linear: An Expressive, Efficient Attention Architecture (Oct 2025)

Title: Kimi Linear: An Expressive, Efficient Attention Architecture (Oct 2025)
Link: http://arxiv.org/abs/2510.26692v1
Date: October 2025

Summary:
Kimi Linear introduces a hybrid linear attention architecture, Kimi Delta Attention (KDA), that extends Gated DeltaNet with a finer-grained gating mechanism and uses specialized Diagonal-Plus-Low-Rank (DPLR) transition matrices for hardware efficiency. The architecture, which interleaves KDA with Multi-Head Latent Attention (MLA) layers, demonstrates superior performance and efficiency compared to full attention models across various tasks, including short-context, long-context, and RL scaling. It significantly reduces KV cache usage by up to 75% and achieves up to 6x faster decoding throughput for 1M contexts, positioning it as a drop-in replacement for full attention.

Key Topics:
Linear Attention
Hybrid Attention Architecture
Kimi Delta Attention (KDA)
Gated DeltaNet
Diagonal-Plus-Low-Rank (DPLR) matrices
Long-context performance
Decoding throughput
KV cache efficiency
Reinforcement Learning (RL)

Chapters:
00:00 - Kimi Linear Overview
00:36 - Introducing KDA & Efficiency
01:14 - Why Address LLM Scaling?
01:52 - Standard Attention Bottlenecks
02:22 - Limitations of Prior Linear Attention
03:10 - KDA: Fine-Grained Memory Control
04:34 - KDA's Implicit Positional Encoding
05:14 - Hardware Optimization: DPLR
07:11 - Hybrid Architecture Strategy
08:43 - No Positional Encoding on MLA
10:27 - Real-World Speed & Efficiency
11:44 - Quality & Benchmark Performance
13:38 - Conclusion & Open-Sourced Code
14:31 - The Future of Positional Encoding?

Stock video credits:
Colin Jones - https://www.pexels.com/@larchmedia
Stas Knop - https://www.pexels.com/@stasknop
Pressmaster - https://www.pexels.com/@pressmaster
Yaroslav Shuraev - https://www.pexels.com/@yaroslav-shuraev
StefWithAnF - https://www.pexels.com/@stefwithanf-1...
Charlie Mounsey - https://www.pexels.com/@charlie-mouns...
@svetjekolem - https://www.pexels.com/@svetjekolem
Kelly - https://www.pexels.com/@kelly
crazy motions - https://www.pexels.com/@crazy-motions...
KATRIN BOLOVTSOVA - https://www.pexels.com/@ekaterina-bol...
Pavel Danilyuk - https://www.pexels.com/@pavel-danilyuk
cottonbro studio - https://www.pexels.com/@cottonbro
Pixabay - https://www.pexels.com/@pixabay
Danil Shostak - https://www.pexels.com/@danil-shostak...
Mikhail Nilov - https://www.pexels.com/@mikhail-nilov
Engin Akyurt - https://www.pexels.com/@enginakyurt
Kindel Media - https://www.pexels.com/@kindelmedia
José Alfredo Munguía Lira - https://www.pexels.com/@rectorretro
Anthony 🙂 - https://www.pexels.com/@inspiredimages
Bedrijfsfilmspecialist.nl - https://www.pexels.com/@bedrijfsfilms...
Pachon in Motion - https://www.pexels.com/@pachon-in-mot...
Silviu Din - https://www.pexels.com/@silviu-din-16...
Oleg Gamulinskii - https://www.pexels.com/@oleg-gamulins...
Trippy Lagoon - https://www.pexels.com/@trippy-lagoon...
Soumya - https://www.pexels.com/@soumya-1446957
Dan Cristian Pădureț - https://www.pexels.com/@paduret