Скачать или смотреть Leave no context behind: Infini attention Efficient Infinite Context Transformers

Ref: https://arxiv.org/pdf/2404.07143
This research introduces Infini-attention, a novel attention mechanism designed to enable large language models (LLMs) to process infinitely long input sequences with bounded memory and computation. Infini-attention integrates compressive memory into the standard attention mechanism, combining masked local attention with long-term linear attention within a single Transformer block. The effectiveness of this approach is demonstrated through experiments on long-context language modeling, passkey retrieval, and book summarization tasks, achieving state-of-the-art results and significant memory compression. The method involves minimal modifications to existing LLMs, facilitating easy integration and adaptation. The research contrasts Infini-attention with other approaches, highlighting its superior performance and efficiency in handling extremely long input sequences.
Infini-attention improves Transformer LLM efficiency for long inputs by combining a compressive memory system with a local attention mechanism in a single Transformer block. This approach enables the model to process theoretically infinite input lengths with limited memory and computational resources.

Advantages of Infini-attention
Bounded Memory Footprint: Unlike traditional Transformers where memory scales quadratically with input length, Infini-attention utilizes a fixed-size compressive memory, ensuring a bounded memory footprint even for very long sequences.
Efficient Streaming Inference: The segment-by-segment processing with a fixed local attention window enables efficient streaming inference, allowing for real-time processing of extremely long inputs.
Continual Pre-training and Adaptation: Infini-attention allows for seamless integration into existing LLMs through continual pre-training. This means models can be adapted to handle long contexts without extensive retraining.
Improved Performance on Long Sequences: Experimental results demonstrate that Infini-attention outperforms baseline models on long-context language modeling, passkey retrieval, and book summarization tasks.
Infini-attention's Effectiveness in Scaling LLMs to Extremely Long Sequences.

Infini-attention enables Transformer LLMs to effectively process extremely long inputs with bounded memory footprint and computation.
It scales LLMs to infinitely long contexts by incorporating a compressive memory into the vanilla attention mechanism. This compressive memory system is more scalable and efficient for handling long sequences compared to the traditional attention mechanism.
Infini-attention combines masked local attention and long-term linear attention mechanisms within a single Transformer block. This allows the model to maintain both global and local context states.
It reuses the key, value, and query states from the standard attention computation for long-term memory consolidation and retrieval. Unlike standard attention, which discards old key-value states, Infini-attention stores these states in the compressive memory.
This approach has minimal impact on existing Transformer architectures, facilitating easy integration through continued pre-training and fine-tuning.
Infini-attention's memory complexity remains constant regardless of input length, unlike other segment-level memory models whose complexity increases with sequence length [6]. It achieves this by storing compressed context in the memory states (Ms and zs) for each head in a single layer. This contrasts with models like Transformer-XL, Compressive Transformer, and Memorizing Transformers that rely on caches that grow with sequence length, or RMT and AutoCompressors that depend on the size of soft-prompt vectors.
Experiments demonstrate Infini-attention's superior performance in long-context language modeling, achieving better perplexity scores while using significantly less memory compared to other models like Memorizing Transformers.
Infini-attention also excels in tasks involving extremely long sequences, such as passkey retrieval with 1M context length and book summarization with 500K length text. It achieves state-of-the-art results on the BookSum dataset, demonstrating its ability to process entire book texts for summarization.

Created with NotebookLM and edited.