Скачать или смотреть Training and Inference on Any-Order Autoregressive Models the Right Way (NeurIPS 2022 oral)

Recent advances in autoregressive modeling have extended beyond fixed sequential generation, enabling any-order inference where variables can be predicted under arbitrary conditioning. This seminar explores the probabilistic foundations and inference mechanisms of Any-Order Autoregressive Models (AO-ARMs), focusing on the discrepancy between training and inference in the univariate conditionals these models encode. While traditional autoregressive models efficiently represent joint distributions through ordered univariate conditionals, they cannot perform arbitrary marginal inference without costly marginalization. AO-ARMs address this by training over all possible orderings, yet face two key challenges: (1) redundant modeling of identical marginals and (2) a mismatch between the distributions of masks used during training and inference. To overcome these issues, the presented paper reinterprets AO-ARMs through a recursive decomposition view, defining a decomposition protocol that determines which index to omit when computing a marginal. Building on this formulation, the proposed Mask-tuned Arbitrary Conditional Model (MAC) introduces (i) lexicographical-edge selection to reduce redundancy and (ii) an induced edge distribution that aligns training with inference by weighting conditionals according to their evaluation frequency. Experiments on language, image, and robotics domains demonstrate the effectiveness of MAC. Overall, this seminar highlights MAC as a principled and efficient framework for modeling arbitrary conditionals—bridging the gap between training objectives and inference behavior in any-order autoregressive systems.

Presenter: Yunseok Han