Скачать или смотреть Probabilistic Inference in Language Models via Twisted Sequential Monte Carlo -- Roger Grosse

Abstract
Numerous capability and safety techniques of Large Language Models (LLMs), including RLHF, automated red-teaming, prompt engineering, and infilling, can be cast as sampling from an unnormalized target distribution defined by a given reward or potential function over the full sequence. In this work, we leverage the rich toolkit of Sequential Monte Carlo (SMC) for these probabilistic inference problems. In particular, we use learned twist functions to estimate the expected future value of the potential at each timestep, which enables us to focus inference-time computation on promising partial sequences. We propose a novel contrastive method for learning the twist functions, and establish connections with the rich literature of soft reinforcement learning. As a complementary application of our twisted SMC framework, we present methods for evaluating the accuracy of language model inference techniques using novel bidirectional SMC bounds on the log partition function. These bounds can be used to estimate the KL divergence between the inference and target distributions in both directions. We apply our inference evaluation techniques to show that twisted SMC is effective for sampling undesirable outputs from a pretrained model (a useful component of harmlessness training and automated red-teaming), generating reviews with varied sentiment, and performing infilling tasks.

Biography
Roger Grosse is an Associate Professor of Computer Science at the University of Toronto, Schwartz-Reisman Chair in Technology and Society, founding member of the Vector Institute for Artificial Intelligence, and a Member of Technical Staff on the Alignment Science team at Anthropic. His research focuses on using our understanding of deep learning to improve the safety and alignment of AI systems. He has held the Sloan Research Fellowship, Canada CIFAR AI Chair, and Canada Research Chair.