Next-Gen AI: RecurrentGemma (Long Context Length)

A brand new Language Model Architecture: RecurrentLLM with Griffin. Moving Past Transformers.
Google developed RecurrentGemma-2B and compares this new LM architecture (!) with the classical transformer based, quadratic complexity of a self-attention Gemma 2B. And the new throughput is: about 6000 tokens per second. Plus two new architectures: GRIFFIN and HAWK, where already HAWK performs better than State Space Models (like Mamba - S6).

Introduction and Model Architecture:
The original paper by Google introduces "RecurrentGemma-2B," leveraging the Griffin architecture, which moves away from traditional global attention mechanisms in favor of a combination of linear recurrences and local attention. This design enables the model to maintain performance while significantly reducing memory requirements during operations on long sequences. The Griffin architecture supports a fixed-size state irrespective of sequence length, contrasting sharply with transformer models where the key-value (KV) cache grows linearly with the sequence length, thereby affecting memory efficiency and speed.

Performance and Evaluation:
RecurrentGemma-2B demonstrates comparable performance to the traditional transformer-based Gemma-2B, despite the former being trained on 33% fewer tokens. It achieves similar or slightly reduced performance across various automated benchmarks, with a detailed evaluation revealing only a marginal average performance drop (from 45.0% to 44.6%). However, the model shines in inference speed and efficiency, maintaining high throughput irrespective of sequence length, which is a notable improvement over transformers.

Technological Advancements and Deployment:
The introduction of a model with such architectural efficiencies suggests potential applications in scenarios where computational resources are limited or where long sequence handling (!) is critical. The team provides tools and code (Github repo, open source) for community engagement, compares the simpler Hawk architecture to state space models (like S4) and also to classical LLama models.

All rights w/ authors of the paper:
RecurrentGemma: Moving Past Transformers for Efficient Open Language Models
https://arxiv.org/pdf/2404.07839.pdf

00:00 Llama 3 inference and finetuning
00:23 New Language Model Dev
01:39 Local Attention
04:22 Linear complexity of RNN
06:05 Gated recurrent unit - GRU
07:56 Linear recurrent Unit - LRU
14:25 GRIFFIN architecture
15:50 Real-Gated Linear recurrent unit RG-LRU
21:20 Griffin Key Features
25:15 RecurrentGemma
26:24 Github code
27:13 Performance benchmark



#ai
#airesearch