Скачать или смотреть Lec 33 | Multimodal Encoder Models

In this lecture, we step beyond text to explore the exciting world of Multimodal Encoder Models, where AI learns to process and understand multiple types of data simultaneously. We begin with the foundational Vision Transformer (ViT), which demonstrated that image patches could be treated just like words in a sentence, and explore early fusion architectures like VisualBERT and ViLBERT that bridge the gap between vision and language. We then discuss CLIP, a game-changing model that learns visual concepts from natural language supervision, enabling powerful zero-shot capabilities.

Moving to more specialized and advanced architectures, we examine LayoutLMv2, which excels at understanding visually rich documents, and address specific architectural improvements like adding Registers to Vision Transformers to remove artifacts. The session also covers the state-of-the-art DINOv3 for self-supervised learning, VideoCLIP for understanding temporal video data, and finally, ImageBind, a unified model that creates a single embedding space to bind together six different modalities—images, text, audio, depth, thermal, and IMU data.

For the presentation slides and other course materials, please visit our main website at lcs2.in/llm2501. For further reading on the concepts discussed, we recommend the following papers:
"An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale" (https://arxiv.org/pdf/2010.11929)
"VisualBERT: A Simple and Performant Baseline for Vision and Language" (https://arxiv.org/pdf/1908.03557)
"ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations" (https://arxiv.org/pdf/1908.02265)
"Learning Transferable Visual Models From Natural Language Supervision" (CLIP) (https://arxiv.org/pdf/2103.00020)
"LayoutLMv2: Multi-modal Pre-training for Visually-rich Document Understanding" (https://arxiv.org/pdf/2012.14740)
"Vision Transformers Need Registers" (https://arxiv.org/abs/2309.16588),
"DINOv3" (https://arxiv.org/abs/2508.10104)
"VideoCLIP" (https://arxiv.org/pdf/2109.14084v2),
"IMAGEBIND: One Embedding Space To Bind Them All" (https://openaccess.thecvf.com/content....

#MultimodalAI #ViT #CLIP #ImageBind #ComputerVision #DeepLearning #LayoutLM #AI #MachineLearning