Predict how your RAG pipeline will handle a query
Most RAG failures are not embedding-quality problems — they are query-type mismatches. A plain similarity search nails “What is our refund window?” but quietly fails on “How does our refund policy compare to our competitor’s, and has it changed since last year?” This analyzer classifies a query and predicts its retrieval difficulty, then tells you which strategy will actually work.
How it works
The tool inspects the query for structural signals: length, the number of distinct entities, conjunctions that hint at multiple sub-questions, comparison and hypothetical phrasing, summarization cues, and temporal dependence. From those signals it assigns a query type and an easy / moderate / hard difficulty rating, then maps the type to a concrete tactic — query decomposition for multi-hop, multi-query expansion for broad asks, HyDE for sparse-match factuals, and reranking for comparisons.
Query types and what they require
Understanding what type of query you are dealing with is the first step to fixing silent retrieval failures. Here is how each type maps to a real challenge:
Factual — a single verifiable fact. Standard similarity search usually works. Difficulty rises only if the fact is phrased very differently from how it appears in the source documents. HyDE (hypothetical document embeddings) can help here: generate a synthetic answer first, then retrieve against that instead of the raw query.
Multi-hop — the answer requires chaining two or more facts from different parts of the corpus. For example, “What is the CEO of the company that acquired Acme in 2022?” Single-vector search rarely handles this well. The correct approach is query decomposition: break the question into “Who acquired Acme in 2022?” and “Who is the CEO of [that company]?”, retrieve for each, then combine.
Comparison — two or more entities are being compared. Retrieval must surface passages about both. Using a single query embedding tends to favour whichever entity appears first or more prominently. Multi-query retrieval — one query per entity, merged before reranking — fixes this.
Hypothetical / counterfactual — “What would happen if…” questions. These rarely match index content directly. Consider converting the hypothetical into a factual form before retrieval, or use a generative step to answer from retrieved principles rather than looking for a direct match.
Summarization — broad “give me an overview” requests. No single chunk contains the answer. Map-reduce retrieval (retrieve many chunks, summarize each, then synthesize) is the standard pattern. Single-pass retrieval produces incomplete answers.
Temporal — the answer depends on when. A metadata date filter narrowing the candidate set before embedding comparison is essential. Without it, an older passage with a high similarity score will silently beat a more recent but slightly less similar one.
Tips for using the analysis
- Route, don’t retrieve blindly. If a query is multi-hop, decompose it into sub-questions and retrieve for each before answering.
- Comparisons need both sides. Make sure your retriever pulls passages for every entity being compared, not just the first one mentioned.
- Temporal queries need metadata filters, not just semantic similarity — filter your index by date before ranking.
- Test your hardest-rated queries first; they expose retrieval gaps fastest.
- Build a small labelled set of ten to twenty representative queries of each type and measure retrieval recall on them. The analyzer shows you where to focus that effort.