Скачать или смотреть Super Weight Ablation Test – Phi 3 Breakdown

This video demonstrates the core finding from Apple’s Super Weights paper: that a tiny number of highly influential parameters—sometimes just a single weight—can disproportionately shape the behavior of a large language model. These “super weights” generate persistent “super activations” that propagate through the network and help maintain coherent token distributions. When they are altered or removed, the model’s internal balance collapses, leading to sharp drops in fluency, correctness, and semantic stability.
In this test, the model is first run with no modifications to establish a clean baseline. Then, super weights identified for Phi-3 are ablated one by one: first (layer 2, row 525, col 808), then adding (1693, 808), and finally (1113, 808). With each additional weight zeroed, the model’s output degrades progressively, until by the third removal it completely loses the ability to form meaningful text. This illustrates just how much leverage these rare parameters exert over the entire forward pass—and why preserving or restoring them during quantization may dramatically improve compressed model quality.

0:00 Model Loading
1:27 Generate first response
1:50 Zero first super-weight and run inference again
2:23 Add second super-weight, forget to zero it, and run inference twice
3:00 Add third weight (of six) and run inference a final time