GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection

Large language models (LLMs) typically demand substantial GPU memory, rendering training impractical on a single consumer GPU, especially for a 7-billion-parameter model that necessitates 58GB of memory. In response, the GaLore paper introduces an innovative strategy that projects gradients into a low-rank space, enabling the model to fit within the constraints of a single GPU. Remarkably, this approach not only addresses the memory challenge but also outperforms other parameter-efficient tuning methods like LoRA, delivering superior results.

paper link: https://arxiv.org/abs/2403.03507

Table of Content:
00:00 Intro
02:17 LoRA
03:18 Limitations of LoRA
05:58 GaLore
18:18 Adam with GaLore
21:01 8-Bit Optimizers
22:50 LOMO
24:48 GaLore vs LoRA
26:20 Rank vs Perplexity
27:07 results

Icon made by Freepik from flaticon.com