Tuning a RAG retriever comes down to one question: how many chunks should I return? Too few and the answer isn’t in the context; too many and you waste tokens and confuse the model. This simulator lets you play out the trade-off with your own scores before touching the pipeline.
How it works
Enter the query, then each candidate chunk with the similarity score your vector search assigned and a checkbox marking whether it is genuinely relevant (your ground truth). The tool sorts the chunks by score — exactly as a real top-k retriever would — and then, for k = 1, 3, 5, and 10, computes:
- Hit@k — was at least one relevant chunk returned?
- Recall@k — what fraction of all relevant chunks made it into the top k?
- Precision@k — what fraction of the top k were relevant?
It also reports the reciprocal rank for this query: 1 divided by the position of the first relevant chunk. Average that across your whole test set and you have MRR.
The recall-precision trade-off in depth
These two metrics move in opposite directions as k increases, and understanding the shape of that trade-off is the core value this simulator provides.
Recall@k answers: “Did the retriever find the answer?” It starts low and climbs as k increases, but eventually plateaus once all relevant chunks are included. Watching where it plateaus tells you the minimum k that captures all the information you need.
Precision@k answers: “How much noise is the LLM reading?” It starts high (if the top result is relevant) and tends to fall as k increases, because you’re including more chunks that happen to score moderately well but don’t actually contain the answer.
The optimal k is where recall is high enough that the answer is almost always present, and precision hasn’t fallen so far that the model is flooded with irrelevant text. This simulator makes that sweet spot visible by showing both metrics across the four standard k values simultaneously.
What MRR and reciprocal rank tell you
Reciprocal rank (RR) is 1 / position-of-first-relevant-chunk. If the first relevant chunk is at position 1 out of 10 results, RR = 1.0. If it’s at position 3, RR = 0.33.
This matters because LLMs do not process all chunks equally. In practice, context near the beginning and end of the input is often weighted more in model attention. A relevant chunk buried at position 9 out of 10 is less useful than one at position 1, even if both nominally appear in the context.
A low average MRR (below 0.4) with high recall@10 is the specific pattern that calls for a re-ranker: your embeddings are retrieving the right chunks, but ranking them poorly. A cross-encoder re-ranker can re-score the top-k candidates and move the most relevant one toward position 1, raising MRR without changing the underlying embedding model.
Diagnosing what the metrics reveal
| Pattern | What it means | What to try |
|---|---|---|
| Low recall@k at all k | Relevant chunks aren’t being retrieved at all | Better chunking, different embedding model, or hybrid sparse+dense retrieval |
| High recall@5, low recall@1 | Answer is there but buried | Add a re-ranker |
| Low precision at low k | Top-ranked chunks are mostly irrelevant | Improve embedding quality or add metadata filtering |
| Recall plateaus at k=3 | You can safely reduce k to 3 | Lower k to save context tokens and cost |
Tips
- Run several representative queries and average the metrics; a single query can be misleading.
- A low MRR with high recall means relevant chunks are retrieved but ranked poorly — a re-ranker is the usual fix.
- Pair this with the RAG Eval Dataset Builder to turn ad-hoc tuning into a repeatable regression test.