Скачать или смотреть Isaac Chung – Reproducibility in Embedding Benchmarks

More: https://2025.berlinbuzzwords.de/sessi...

Speaker: Isaac Chung

Reproducibility in embedding benchmarks is challenging, especially with embedding models that are instruction-tuned and increasingly large. Learn how MTEB tackles prompt variability, scaling issues, and large datasets to ensure fair and consistent evaluations, setting a standard for benchmarking in embeddings.

Reproducibility in embedding benchmarks is no small feat. Prompt variability, growing computational demands, and evolving tasks make fair comparisons a challenge. The need for robust benchmarking has never been greater.

The Massive Text Embedding Benchmark (MTEB) addresses these challenges with a standardized, open-source framework for evaluating text embedding models. Covering diverse tasks like clustering, retrieval, and classification, MTEB ensures consistent and reproducible results. Extensions like MMTEB (multilingual) and MIEB (image) further expand its capabilities.

In this talk, we’ll explore the quirks and complexities of benchmarking embedding models, such as prompt sensitivity, scaling issues, and emergent behaviors. We’ll show how MTEB simplifies reproducibility, making it easier for researchers and industry practitioners to measure progress, choose the right models, and push the boundaries of embedding performance.

###

Follow us on Social Media and join the Community!

Mastodon: https://floss.social/@BerlinBuzzwords
LinkedIn:   / berlin-buzzwords  
Website: https://berlinbuzzwords.de
Mail: [email protected]

Berlin Buzzwords is an event by Plain Schwarz – https://plainschwarz.com