Скачать или смотреть 【YouTube動画で学習】ロボットが"人間の記憶"を獲得、家事指示が通じる時代へ

論文情報
・url: http://arxiv.org/html/2602.23937v1
・title: Enhancing Vision-Language Navigation with Multimodal Event Knowledge from Real-World Indoor Tour Videos
・abstract: Vision-Language Navigation (VLN) agents often struggle with long-horizon reasoning in unseen environments, particularly when facing ambiguous, coarse-grained instructions. While recent advances use knowledge graph to enhance reasoning, the potential of multimodal event knowledge inspired by human episodic memory remains underexplored. In this work, we propose an event-centric knowledge enhancement strategy for automated process knowledge mining and feature fusion to solve coarse-grained instruction and long-horizon reasoning in VLN task. First, we construct YE-KG, the first large-scale multimodal spatiotemporal knowledge graph, with over 86k nodes and 83k edges, derived from real-world indoor videos. By leveraging multimodal large language models (i.e., LLaVa, GPT4), we extract unstructured video streams into structured semantic-action-effect events to serve as explicit episodic memory. Second, we introduce STE-VLN, which integrates the above graph into VLN models via a Coarse-to-Fine Hierarchical Retrieval mechanism. This allows agents to retrieve causal event sequences and dynamically fuse them with egocentric visual observations. Experiments on REVERIE, R2R, and R2R-CE benchmarks demonstrate the efficiency of our event-centric strategy, outperforming state-of-the-art approaches across diverse action spaces. Our data and code are available on the project website https://sites.google.com/view/y-event....
==========

お掃除ロボットに「冷蔵庫の横の青い箱を取ってきて」と頼める日が来るかもしれません。最新AI研究では、YouTubeの室内ツアー動画320時間分から「人間のような記憶」を抽出し、ロボットに教え込むことに成功。曖昧な指示でも理解できる次世代ナビゲーション技術が、介護・物流・オフィスの現場を変えようとしています。5年後、あなたの生活はどう変わる？