Скачать или смотреть Joint Sample+Token Pruning for LLM SFT

In this AI Research Roundup episode, Alex discusses the paper:
'Winning the Pruning Gamble: A Unified Approach to Joint Sample and Token Pruning for Efficient Supervised Fine-Tuning'
The authors tackle compute-heavy supervised fine-tuning by jointly pruning samples and tokens instead of treating them separately. They frame pruning as a bilevel optimization that selects per-step sample and token pruners under target keep ratios. Using an Error–Uncertainty plane, training data is categorized into four quadrants to drop noisy/redundant examples and keep valuable or calibrating ones. Their two-stage Quadrant-based Tuning dynamically finds PPL/entropy thresholds to retain useful samples and applies asymmetric, context-aware token pruning specifically to confidently-wrong (Q2) cases.
Paper URL: https://arxiv.org/abs/2509.23873

#AI #MachineLearning #DeepLearning #LLM #SupervisedFineTuning #DataPruning #TokenPruning #BilevelOptimization