Скачать или смотреть Visual Document Retrieval: Clustering, Search, and Bias Detection in Image Datasets Using Embeddings

In this talk you'll learn about the task of visual document retrieval, the models which are widely used by the community, and see them in action through the open source FiftyOne App where you'll learn how to use these models to identify groups and clusters of documents, find unique documents, uncover biases in your visual document dataset, and search over your document corpus using natural language.

GitHub Repo - https://github.com/harpreetsahota204/...

About the Speaker

Harpreet Sahota is a hacker-in-residence and machine learning engineer with a passion for deep learning and generative AI. He’s got a deep interest in VLMs, Visual Agents, Document AI, and Physical AI.

00:00 - Intro
00:22 - Overview of Visual Document Retrieval
01:22 - Presenter Background and Session Goals
02:07 - Why Traditional Text Search Fails
03:08 - Documents as Visual Artifacts
03:45 - Limitations of CLIP and Similar Models
04:46 - Natural vs. Document Images
05:58 - Specialized Models for Document Retrieval
06:53 - Spatial Structure and Multi-Vector Queries
07:58 - Pattern Recognition in Document Embeddings
08:40 - Comparison with OCR Pipelines
09:59 - Benefits of Semantic vs. Keyword Search
10:54 - Integrated Models in FiftyOne
11:01 - What You Can Do With Document Embeddings
11:18 - Four Key Analyses with Embeddings
11:25 - Embedding Space Visualization
12:21 - Measuring Uniqueness in Datasets
13:37 - Semantic Similarity Search
14:58 - Understanding Representativeness Scores
16:19 - Embeddings Enable Data Understanding at Scale
17:03 - Use Case: Exploring NeurIPS Papers
18:06 - Workflow Demo: From Dataset to Discovery
19:00 - Loading and Preparing the Visual AI Dataset
20:13 - Installing Models and Setting Up Environment
21:00 - Previewing and Exploring Samples in FiftyOne
22:06 - Filtering by Research Categories
23:38 - Abstract Keyword Search Demo
24:20 - Using Tags to Mark Papers of Interest
25:01 - Model Setup and Available Options
26:05 - Computing Embeddings and Visualizing Them
27:10 - Creating and Using Similarity Indexes
28:03 - Computing Uniqueness and Detecting Duplicates
29:33 - Understanding Representativeness in Practice
30:14 - Zero-Shot Classification of Research Papers
31:22 - Exploring Embedding Visualizations in the App
32:20 - Clustering Similar Research Papers
33:03 - Semantic Similarity Search on a Sample Paper
34:41 - Evaluating Model Similarity Results
35:24 - Summary of Visualization and Filtering Features
35:42 - Evaluating Classification Accuracy
36:45 - Access to Resources, Notebooks, and GitHub Repo
37:28 - Closing and Next Steps