Скачать или смотреть Orion-MSP: Sparse Multi-Scale Tabular ICL

In this AI Research Roundup episode, Alex discusses the paper:
'Orion-MSP: Multi-Scale Sparse Attention for Tabular In-Context Learning'
The authors propose a table-native transformer that tackles single-scale processing and quadratic feature attention in tabular ICL. Orion-MSP models hierarchical feature interactions via multi-scale processing, and swaps dense attention for structured block-sparse attention to achieve near-linear complexity. A Perceiver-style latent memory enables safe bidirectional communication among components while preserving ICL safety. The architecture includes column-wise Set-Transformer embeddings, multi-scale sparse row interactions, and a split-masked predictor with label injection.
Paper URL: https://arxiv.org/abs/2511.02818

#AI #MachineLearning #DeepLearning #TabularData #InContextLearning #Transformer #SparseAttention #Perceiver

Resources:
Hugging Face model: https://huggingface.co/Lexsi/Orion-MSP