Training Multi-Modal AI: Inside the Jina CLIP Embedding Model | S2 E11

Today we are talking to Michael Günther, a senior machine learning scientist at Jina about his work on JINA Clip.

Some key points:

*Uni-modal embeddings* convert a single type of input (text, images, audio) into vectors
*Multimodal embeddings* learn a joint embedding space that can handle multiple types of input, enabling cross-modal search (e.g., searching images with text)
Multimodal models can potentially learn richer representations of the world, including concepts that are difficult or impossible to put into words

Types of Text-Image Models

1. *CLIP-like Models*
Separate vision and text transformer models
Each tower maps inputs to a shared vector space
Optimized for efficient retrieval
2. *Vision-Language Models*
Process image patches as tokens
Use transformer architecture to combine image and text information
Better suited for complex document matching
3. *Hybrid Models*
Combine separate encoders with additional transformer components
Allow for more complex interactions between modalities
Example: Google's Magic Lens model

Training Insights from Jina CLIP

1. *Key Learnings*
Freezing the text encoder during training can significantly hinder performance
Short image captions limit the model's ability to learn rich text representations
Large batch sizes are crucial for training embedding models effectively
2. *Training Process*
Three-stage training approach:
Stage 1: Training on image captions and text pairs
Stage 2: Adding longer image captions
Stage 3: Including triplet data with hard negatives

Practical Considerations

*Similarity Scales*
Different modalities can produce different similarity value scales
Important to consider when combining multiple embedding types
Can affect threshold-based filtering
*Model Selection*
Evaluate models based on relevant benchmarks
Consider the domain similarity between training data and intended use case
Assessment of computational requirements and efficiency needs

Future Directions

1. *Areas for Development*
More comprehensive benchmarks for multimodal tasks
Better support for semi-structured data
Improved handling of non-photographic images
2. *Upcoming Developments at Jina AI*
Multilingual support for Jina ColBERT
New version of text embedding models
Focus on complex multimodal search applications

Practical Applications

*E-commerce*
Product search and recommendations
Combined text-image embeddings for better results
Synthetic data generation for fine-tuning
*Fine-tuning Strategies*
Using click data and query logs
Generative pseudo-labeling for creating training data
Domain-specific adaptations

Key Takeaways for Engineers

1. Be aware of similarity value scales and their implications
2. Establish quantitative evaluation metrics before optimization
3. Consider model limitations (e.g., image resolution, text length)
4. Use performance optimizations like flash attention and activation checkpointing
5. Universal embedding models might not be optimal for specific use cases

*Michael Guenther*

[LinkedIn](https://www.linkedin.com/in/michael-g...)
[X (Twitter)](https://x.com/michael_g_u)
[Jina AI](https://jina.ai/)
[New Multilingual Embedding Modal](https://jina.ai/news/jina-embeddings-...)

*Nicolay Gerold:*

[⁠LinkedIn⁠](  / nicolay-gerold  )
[⁠X (Twitter)](  / nicolaygerold  )

00:00 Introduction to Uni-modal and Multimodal Embeddings
00:16 Exploring Multimodal Embeddings and Their Applications
01:06 Training Multimodal Embedding Models
02:21 Challenges and Solutions in Embedding Models
07:29 Advanced Techniques and Future Directions
29:19 Understanding Model Interference in Search Specialization
30:17 Fine-Tuning Jina CLIP for E-Commerce
32:18 Synthetic Data Generation and Pseudo-Labeling
33:36 Challenges and Learnings in Embedding Models
40:52 Future Directions and Takeaways