Скачать или смотреть VaultGemma 1B: Differentially Private Language Model Training and Scaling, prioritizing privacy LLM

VaultGemma 1B: Differentially Private Language Model Training and Scaling (LLM)

The research introduces VaultGemma 1B, a novel 1-billion-parameter language model, distinguished by its complete training with differential privacy from its inception, a significant step in developing privacy-preserving AI. This approach directly addresses the challenge of memorization of sensitive training data often found in large language models by applying rigorous mathematical frameworks from the outset, rather than only during fine-tuning. The paper also details the architectural choices, training methodologies, and evaluation processes that led to VaultGemma, including the use of specific attention mechanisms and pre-normalization techniques. While acknowledging a present utility gap compared to non-private models, the authors present scaling laws for differentially private models, indicating a path forward for future improvements. Ultimately, the release of VaultGemma aims to foster further research and development in private AI, offering a robust foundation for applications prioritizing data privacy.