Скачать или смотреть [DL Math+Efficiency] Flavio Martinelli - Flat Channels to Infinity in Neural Loss Landscapes

Title: Flat Channels to Infinity in Neural Loss Landscapes

Abstract: The loss landscapes of neural networks contain minima and saddle points that may be connected in flat regions or appear in isolation. We identify and characterize a special structure in the loss landscape: channels along which the loss decreases extremely slowly, while the output weights of at least two neurons, a_i and a_j, diverge to ±infinity, and their input weight vectors, w_i and w_j, become equal to each other. At convergence, the two neurons implement a gated linear unit: a_iσ(w_i · x) + a_jσ(w_j · x) → cσ(w · x) + (v · x)σ′(w · x). Geometrically, these channels to infinity are asymptotically parallel to symmetry-induced lines of critical points. Gradient flow solvers, and related optimization methods like SGD or ADAM, reach the channels with high probability in diverse regression settings, but without careful inspection they look like flat local minima with finite parameter values. Our characterization provides a comprehensive picture of these quasi-flat regions in terms of gradient dynamics, geometry, and functional interpretation. The emergence of gated linear units at the end of the channels highlights a surprising aspect of the computational capabilities of fully connected layers.

arXiv: https://arxiv.org/pdf/2506.14951

DL Math&Efficiency homepage: https://sites.google.com/view/efficie...