Скачать или смотреть CLIMB: CLustering-based Iterative Data Mixture Bootstrapping (Apr 2025)

Title: CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training (Apr 2025)
Link: http://arxiv.org/abs/2504.13161v1
Date: April 2025

Summary:
The paper introduces CLIMB, a framework for automated data mixture optimization in language model pre-training. CLIMB uses clustering and iterative search to find optimal data combinations, improving performance and domain adaptation. Experiments show CLIMB outperforms existing methods, achieving state-of-the-art results with a 1B model. The paper also releases ClimbLab and ClimbMix, new datasets for research.

Key Topics:
Data Mixture
Pre-training
Clustering
Iterative Bootstrapping
Language Models
Domain Adaptation

Chapters:
00:00 - Introduction to CLIMB
00:16 - The Data Problem
00:30 - Climix Dataset
01:10 - The Data Challenge
02:11 - CLIMB Framework Overview
02:19 - Data Preprocessing
02:53 - Clustering
03:14 - Merging and Pruning
04:17 - Iterative Bootstrapping
04:37 - Bi-level Optimization Explained
04:55 - Predictor Model
05:12 - Proxy Models
05:54 - Search Focus
06:10 - CLIMB Framework Steps
06:50 - Performance Results
07:26 - Generalization
07:46 - Climix Performance
08:40 - Domain Optimization
09:37 - Data Isn't Universal
10:00 - Ablation Studies
10:47 - Number of Clusters
11:08 - The Starting Point
11:38 - Cluster Weights
12:31 - Climbing Helps Navigate
12:47 - Climix and Clim Lab
13:46 - Core Innovation