Скачать или смотреть Transformers without Normalization using Dynamic Tanh (DyT)

Transformers without Normalization using Dynamic Tanh (DyT)

The tutorial provides an introduction to Dynamic Tanh (DyT), a simple element-wise operation designed to replace normalization layers in Transformer neural network architectures. The research claims that DyT, defined as $\text{tanh}(\alpha x)$ where $\alpha$ is a learnable parameter, can achieve performance equal to or better than traditional normalization methods like Layer Normalization (LN) or RMSNorm across diverse tasks, including language modeling and computer vision. This is inspired by the observation that normalization layers often produce a tanh-like, S-shaped input-output mapping, which DyT directly models. The findings suggest that normalization layers, long considered essential for stable training, may not be indispensable if replaced by a suitable alternative like DyT.