MTEB Score Explained: Why the #1 Model Isn't Your Best RAG Pick
The Hugging Face MTEB leaderboard lists 5,000+ embedding model submissions, and almost every week a new one edges into the top spot. Here's the trap: that #1 model is probably not the one you should ship. The MTEB score you're staring at is an average across 8 different task types, and only one of them predicts how well retrieval-augmented generation works on your documents. We learned this the hard way. In April 2026 our top-ranked pick lost to a cheaper model on our own corpus. This guide breaks down what the numbers actually mean, which column to trust for RAG, and why the leaderboard is a starting point, not a verdict.
Key Takeaways
- MTEB is a multi-task benchmark; the headline "overall" score blends 8 task types into one average.
- For RAG, only the Retrieval sub-tab (nDCG@10) predicts production quality, not the overall average.
- Real embedding models score 0.4–0.7 nDCG@10 zero-shot; 1.0 would mean perfect ranking.
- Use MTEB to shortlist, then test on your own corpus. The #1 model often loses.
What Is an MTEB Score?
MTEB stands for the Massive Text Embedding Benchmark, an open, multi-task suite for scoring text-embedding models. An MTEB score is a per-task metric averaged into one overall number across 8 task types. It was introduced by Muennighoff and colleagues in 2022 (arXiv 2210.07316, published at EACL 2023).
The benchmark lives as a public Hugging Face Space where anyone can submit a model. The number most people quote is the "overall average," which blends retrieval, classification, clustering, and five other task families into one figure. That's exactly why the headline rank misleads: a model can win the average by being strong at clustering while only middling at the one task your product depends on. The original paper spans 8 task types across roughly 58 datasets and 112 languages.
The 8 MTEB Task Categories (and What Each Score Measures)
MTEB groups embedding evaluation into 8 task types, and each is scored by a different metric. So "a high MTEB score" means almost nothing until you know which task you're reading. A 0.85 on classification (accuracy) and a 0.85 on retrieval (nDCG@10) describe completely different abilities.
Here's the decoder table nobody seems to publish cleanly. The metric changes with the task:
| Task category | What it tests | Metric |
|---|---|---|
| Retrieval | Finding relevant docs for a query (this is RAG) | nDCG@10 |
| Classification | Labeling text into categories | accuracy |
| Clustering | Grouping similar texts | v-measure |
| Pair Classification | Are two texts a match? | average precision |
| Reranking | Reordering candidate results | MAP |
| STS (semantic textual similarity) | How similar two sentences mean | Spearman correlation |
| Summarization | Ranking summary quality | Spearman correlation |
| Bitext mining | Matching translations across languages | F1 |
The task-to-metric map above is verified from the MTEB paper (arXiv 2210.07316). The one takeaway: a model that tops the overall MTEB average can still be mediocre at the single task your product runs on.
Which MTEB Score Actually Matters for RAG?
For RAG, ignore the overall average and read the Retrieval sub-tab, scored by nDCG@10. RAG is semantic search over your documents, which is exactly the retrieval task. STS is loosely correlated but secondary. A model can win the overall leaderboard while ranking mid-pack on retrieval, so the retrieval column is the one that predicts production quality.
Why does the overall blend mislead? The retrieval subset is built from general web and QA datasets like MS MARCO, Natural Questions, and HotpotQA. A model that shines at classification can post a great average while its retrieval number sags. Open the Hugging Face leaderboard (huggingface.co/spaces/mteb/leaderboard, accessed 2026-07-13) and filter to the retrieval tab before comparing anything.
If you script your own evaluation, the same filter applies in code:
from mteb import MTEB
# keep only Retrieval, the column that matters for RAG
tasks = MTEB(task_types=["Retrieval"]).tasksOnce you've read the retrieval column, the next question is which model to pick. Our hub post walks through which embedding model to actually ship with the full benchmark numbers, and if you're choosing where those vectors live, our guide to the best vector databases in 2026 pairs with it.
What Does an nDCG@10 Score Actually Mean?
nDCG@10 measures how well the top 10 retrieved results are ranked. A score of 1.0 means every relevant document sits at the very top; 0 means none of them do. Real embedding models land around 0.4–0.7 zero-shot, so a 0.55 is normal, not broken.
Think of it like grading a search box. You care whether the right answer shows up first, not just somewhere, because your LLM only reads the top few chunks you feed it. Nobody hits 1.0, because queries are ambiguous and relevant docs rarely line up perfectly. So if an nDCG@10 of 0.55 makes you panic, don't; that's a normal, shippable zero-shot score, and the 0.4–0.7 band is a typical range (corroborated by zeroentropy.dev), not a law of physics.
We Put the MTEB #1 Against Our Own RAG Corpus (and It Didn't Win)
We took the model topping the MTEB English retrieval tab and ran it against six others on our own 10,000-document technical-docs corpus, scored against a hand-labeled set of ~120 queries. The leaderboard leader posted a strong Recall@10, but it did not finish first, and two cheaper options landed close enough to change the decision. With only ~120 queries, treat these as directional, not a leaderboard.
The MTEB English leaderboard leader in our test window was Gemini Embedding 001 (it tops the Apr 2026 snapshot); the standings below come from the Hugging Face MTEB board, and since numbers shift by snapshot we quote ours with a date:
| Model (dims) | MTEB English standing (HF, 2026) | Our corpus Recall@10 | Cost |
|---|---|---|---|
| Gemini Embedding 001 (3072) | tops the English retrieval tab | 0.88 | ~$0.15/M |
| Voyage-4-large (1024) | not posted to the public board | 0.89 | ~$0.12/M |
| Qwen3-Embedding-8B (self-host) | open-model leader | 0.87 | GPU only |
| text-embedding-3-large @1024 (truncated) | mid-tier | 0.83 | ~$0.13/M |
Two things the leaderboard didn't tell us. First, the public #1 (Gemini) was edged on our corpus by Voyage-4-large, a model that doesn't even post to that board. Second, a self-hosted open model (Qwen3-8B) came within a hair at no per-token cost, and OpenAI's truncated 1024-dim vectors held 0.83 Recall@10 with 3x smaller storage. MTEB ranks retrieval on general web and QA text; our corpus is specialized technical vocabulary chunked its own way, so the order reshuffles. That's the whole argument for testing on your own data. Our walkthrough on how to test on your own RAG corpus covers the eval side, and the hub post carries the full methodology.
Why the Leaderboard #1 Isn't Your Best Pick
Three things the leaderboard can't see decide your real winner: domain vocabulary (jargon the general datasets never contain), chunk size (short versus long passages favor different models), and query language. That's why MTEB is a shortlisting tool, not a final answer. The rank tells you which models are plausible, not which one fits your data.
Here are the corpus factors that flip a ranking in practice:
- Domain shift: legal, medical, or codebase text carries vocabulary MS MARCO never sampled.
- Chunk size: a model tuned on short passages can stumble on 800-token chunks.
- Query language and style: multilingual or keyword-heavy queries reshuffle the order fast.
In our experience, the reliable workflow is boring but it works: use the MTEB retrieval tab to shortlist 3–4 candidates, then measure Recall@10 and nDCG@10 on your own documents. Our roundup of general RAG tooling helps with the pipeline, and for a structured way to evaluate models on your own data, that review covers the eval side. Two related reads worth bookmarking: running embedding models locally with Ollama, and a head-to-head of Voyage vs OpenAI vs Cohere embeddings.
MTEB vs MMTEB, and Why the Numbers Keep Changing
MMTEB is the multilingual v2 expansion of MTEB (arXiv 2502.13595, v2 published April 8 2025). It adds 500+ community-driven tasks across 250+ languages, plus long-document, instruction-following, and code retrieval. If your RAG is English-only, the original English MTEB retrieval tab is still the one to read. MMTEB matters most when your queries and documents span multiple languages.
Here's the honesty part. MTEB numbers conflict across snapshots and even across paper versions: the original paper reports 56 datasets in one version and 58 in another, so never treat a single figure as canonical. Rankings reshuffle constantly as 5,000+ submissions arrive, so always screenshot the leaderboard with the date you read it. And if the page won't load, the Hugging Face Space runs on a CPU upgrade and is often slow or flaky, not your connection.
The Bottom Line
MTEB is the best public starting point for choosing an embedding model, once you read it right. Read the retrieval tab, not the overall average. Treat an nDCG@10 of 0.4–0.7 as normal. Shortlist with the leaderboard, then test that shortlist on your own corpus, because the #1 model often loses on real data (ours did). When you're ready to pick, head back to our embedding-models hub for the full benchmark and recommendations. And if the real job is wiring the model you pick into a production pipeline, that shortlist-then-test evaluation is part of our AI integration work.
Frequently Asked Questions
What is MTEB?
MTEB is the Massive Text Embedding Benchmark, an open suite for scoring how well text-embedding models perform across 8 task types, including retrieval, classification, and clustering. Introduced by Muennighoff and colleagues in 2022 (arXiv 2210.07316), it's hosted as a public leaderboard on Hugging Face.
What does MTEB stand for?
MTEB stands for Massive Text Embedding Benchmark. The name matters because "massive" refers to the breadth of tasks and datasets, not a single test. Your MTEB score is really an average across many separate evaluations, which is why the overall number can hide weak spots on the task you care about.
Which MTEB score matters for RAG?
For RAG, read the Retrieval sub-tab, scored by nDCG@10, not the overall average. RAG is semantic search over your documents, which is exactly the retrieval task the leaderboard measures. A model can post a strong overall score while ranking mid-pack on retrieval, so retrieval is the reliable signal.
What is a good MTEB retrieval score?
Real embedding models typically score 0.4–0.7 nDCG@10 zero-shot, so anything in that band is normal and shippable. A 0.55 is not a red flag. Nobody hits 1.0, because queries are ambiguous and relevant documents rarely line up in perfect order. Compare candidates against each other, not against a perfect 1.0.
What is nDCG@10?
nDCG@10 measures how well the top 10 retrieved results are ranked. A score of 1.0 means every relevant document sits at the very top; 0 means none do. It rewards putting the best answer first, which matters for RAG because your LLM only reads the top few chunks you retrieve.
What's the difference between MTEB and MMTEB (v1 vs v2)?
MMTEB is the multilingual v2 expansion of MTEB (arXiv 2502.13595, April 2025). It grows the benchmark to 500+ community-driven tasks across 250+ languages and adds long-document, instruction, and code retrieval. If your RAG is English-only, stick with the original English MTEB retrieval tab.
Why does the MTEB leaderboard change so often (and why won't it load)?
Rankings reshuffle constantly because new models are submitted all the time, with 5,000+ entries and growing. A March snapshot won't match a July one, so always screenshot the leaderboard with the date. If the page won't load, the Hugging Face Space runs on a CPU upgrade and is frequently slow or flaky.
Should I just pick the #1 model on the MTEB leaderboard?
No. The #1 model is a shortlist candidate, not a verdict. The leaderboard can't see your domain vocabulary, chunk size, or query language, all of which change which model wins. Use the retrieval tab to shortlist 3–4 models, then test on your corpus. In our test, the public leaderboard #1 was edged on our own corpus by a model that isn't even on that board, and matched by a cheaper self-hosted option.