Скачать или смотреть Real-Time Chunking (RTC) for Seamless AI Robot Control

*Real-Time Chunking (RTC)* is a novel, essential inference-time algorithm that solves the critical problem of latency in modern robotics, allowing large Vision-Language-Action (VLA) foundation models like $\pi^0$ and $\pi^{0.5}$ to perform smooth, continuous, and highly reactive movements.

Modern VLA policies often have billions of parameters, leading to high inference latency—frequently exceeding 100 milliseconds or more. While robots use *Action Chunking* (generating a sequence of future actions, often 50 steps at a time) to amortise this cost, a naive execution strategy—like the traditional synchronous inference approach—results in disruptive pauses, out-of-distribution jerky movements, or sudden strategy changes whenever the robot waits for the next action chunk.

RTC enables *asynchronous execution* (or "thinking while moving"), allowing the VLA policy to continuously generate the next action chunk while the robot executes the current one, effectively eliminating the pauses that slow down execution.

#### How RTC Works: The Inpainting Solution

RTC’s core insight is to treat the transition between action chunks as an **inpainting problem**. It ensures seamless continuity by:

1. *Estimating Inference Delay:* The system estimates the expected delay (d) and defines a *frozen prefix* of actions guaranteed to execute from the previous chunk.
2. *Guided Flow Matching:* The new action chunk is generated using a diffusion- or flow-based VLA policy. This process is conditioned on the frozen prefix to ensure the newly planned trajectory aligns with the robot’s ongoing motion.
3. *Soft Masking:* A *soft masking* technique is applied to overlapping time steps immediately following the frozen prefix. This soft constraint, typically using exponential decay, guides the new prediction toward the executing path while simultaneously allowing the model to incorporate the latest sensory observations to adapt its strategy dynamically.

RTC is robust to high inference delays, successfully handling precise and dynamic tasks—such as *lighting a match* or plugging in an Ethernet cable—even with latencies over **300 milliseconds**. This leads to significantly improved task throughput and fewer mistakes compared to synchronous or temporal ensembling methods. Critically, RTC is an inference-time method and requires **no changes to the original model’s training process**.