Скачать или смотреть Matryoshka Quantization: Training Once, Serving at Any Precision

The paper introduces Matryoshka Quantization (MatQuant),
a novel multi-scale quantization method designed to optimize large models for multiple precision levels (int8, int4, and int2) within a single trained model. Unlike conventional quantization approaches that require separate models for each bit-width, MatQuant leverages the inherent nested structure of integer data types, enabling a single model to serve different precision levels dynamically. This eliminates the need for multiple quantized models while improving int2 accuracy by up to 10% compared to traditional techniques like Quantization Aware Training (QAT) and OmniQuant. Additionally, MatQuant allows interpolative bit-width extraction (e.g., int3, int6) and Mix’n’Match strategies, enabling fine-grained control over accuracy and efficiency trade-offs during deployment. These innovations make MatQuant a highly flexible and efficient quantization framework, pushing the boundaries of low-bit inference for large models.

find paper here: https://arxiv.org/pdf/2502.06786v1
#llm #llmops #quantization #deepmind #google