Where rerankers actually help — and where they don't

The seductive promise of reranking

Cross-encoder rerankers have become the default recommendation in RAG (retrieval-augmented generation) architectures. The pitch is compelling: "Your vector embeddings miss semantic nuances. Add a reranker to catch them."

In theory, rerankers re-score retrieved documents using a more sophisticated cross-encoder model that sees both query and document simultaneously — unlike bi-encoders that embed them separately.

The promise: Better precision with minimal effort.

The reality: It depends.

Experimental observations across production systems

Consider reranker behavior across four different production RAG scenarios observed over extended periods:

1. Legal contract search (50K documents, 2K daily queries)

2. Technical documentation (200K documents, 5K daily queries)

3. Customer support KB (15K documents, 10K daily queries)

4. Research paper discovery (1M documents, 500 daily queries)

A/B testing patterns compared:

• Bi-encoder retrieval alone (BGE-M3).

• Bi-encoder + cross-encoder reranker (BGE-reranker-v2-m3).

Key metrics tracked:

• Recall@K: Percentage of relevant documents in top K results.

• MRR (Mean Reciprocal Rank): Position of first relevant result.

• Latency: P50, P95, P99 response times.

• Cost: Infrastructure and compute expenses.

When rerankers provide measurable value

Use case 1: Ambiguous or multi-intent queries

Example query: "What's our return policy?"

This query could mean:

• Return policy for physical products.

• Return policy for digital downloads.

• Return policy for damaged items.

• Refund timelines.

Bi-encoder embeddings often conflate these nuances. Rerankers can disambiguate by examining query-document pairs holistically.

Real data (customer support KB):

Metric	Bi-encoder only	+ Reranker
Recall@5	76%	89%
MRR	0.64	0.81
Latency (p95)	95ms	245ms

Observation: +13% recall improvement at the cost of 2.5x latency increase proves worthwhile in support contexts where precision matters more than speed.

Use case 2: Long-form documents with subtle relevance signals

Scenario: Legal contract search where relevance hinges on specific clauses buried in 50-page documents.

Bi-encoders compress entire documents (or chunks) into fixed-size vectors, losing fine-grained details. Rerankers can re-score based on exact clause matches.

Real data (legal contract search):

Metric	Bi-encoder only	+ Reranker
Recall@10	82%	94%
False positive rate	18%	7%
Latency (p95)	180ms	420ms

Implementation detail:

from sentence_transformers import CrossEncoder
reranker = CrossEncoder('BAAI/bge-reranker-v2-m3', max_length=1024)
Retrieve 50 candidates with bi-encoder
candidates = vector_db.search(query, limit=50)
Rerank to top 10 with cross-encoder
pairs = [[query, doc.text] for doc in candidates]
scores = reranker.predict(pairs)
Sort by reranker scores
reranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)[:10]

Observation: Essential pattern for precision-critical domains where false positives carry significant business cost. The latency penalty remains acceptable for async workflows.

Use case 3: Queries with domain-specific terminology

Example: "What's the pharmacokinetics of metformin in CKD Stage 3 patients?"

Generic bi-encoders struggle with specialized vocabulary. Rerankers trained on domain-specific data can better assess relevance.

Real data (research paper discovery):

Metric	Bi-encoder only	BGE-M3 + Reranker	Fine-tuned reranker
Recall@10	68%	79%	88%
Latency (p95)	220ms	410ms	450ms

Key insight: Fine-tuning the reranker on domain-specific data provides additional recall improvements beyond generic models.

When rerankers add cost without value

Anti-pattern 1: Simple factoid queries

Example: "What is Kubernetes?"

For straightforward queries with clear intent, bi-encoders already perform well. Reranking adds latency without improving results.

Real data (technical documentation):

Query type	Recall@5 (bi-encoder)	Recall@5 (+ reranker)	Latency increase
Factoid ("What is X?")	94%	95%	+130ms
Navigational ("X documentation")	97%	97%	+140ms

Observation: For simple queries with clear intent, the 1-3% improvement doesn't justify the latency penalty.

Anti-pattern 2: Small candidate sets

Rerankers shine when discriminating among many candidates. With small candidate sets (<10 documents), the improvement is negligible.

Experiment: Rerank 5 vs. 50 candidates

Candidates retrieved	Recall@5 improvement	Latency overhead
5	+1.2%	+85ms
20	+6.5%	+150ms
50	+11.8%	+220ms

Recommendation: Retrieve at least 20 candidates before reranking. Otherwise, improve your bi-encoder embedding model instead.

Anti-pattern 3: Real-time conversational interfaces

Chat interfaces demand sub-200ms latency. Rerankers typically add 100-300ms, making interactions feel sluggish.

User experience threshold:

• <100ms: Instant.

• 100-300ms: Slight delay (acceptable for search).

• 300-1000ms: Noticeable lag (frustrating for chat).

• >1000ms: Broken experience.

Real data (customer support chat):

Configuration	Latency (p95)	User satisfaction
Bi-encoder only	110ms	4.2/5
+ Reranker (async)	180ms	4.1/5
+ Reranker (sync)	380ms	3.6/5

Observation: For chat interfaces, latency thresholds directly impact user experience more than marginal recall improvements.

Cost analysis: The hidden overhead

Rerankers aren't just slow — they're expensive at scale.

Compute costs

Assumptions:

• 1M queries/month.

• 20 candidates per query.

• Reranker: BGE-reranker-v2-m3 (560M parameters).

Infrastructure:

Component	Without reranker	With reranker
GPU instances	2x T4 ($200/mo)	4x T4 ($400/mo)
Inference latency	120ms	350ms
Monthly cost	$200	$400

Cost per 1M queries:

• Bi-encoder only: $200.

• + Reranker: $400.

Break-even analysis: Reranking must improve business outcomes by at least $200/month to justify the cost.

Latency budget allocation

Every millisecond counts in user-facing systems. Here's how latency breaks down in a typical RAG pipeline:

Total query latency: 850ms
├─ Embedding generation: 50ms
├─ Vector search: 80ms
├─ Reranking: 200ms           ← 24% of total latency
├─ LLM inference: 450ms
└─ Network overhead: 70ms

Reranking consumes 24% of the latency budget for marginal recall improvements. Consider whether that budget is better spent on:

• Faster LLM inference (quantization, better GPUs).

• Improved chunking strategies.

• Better embedding models.

Decision framework: Should you use a reranker?

Use a reranker if:

✅ Precision is critical (legal, medical, compliance domains)

• False positives are expensive.

• User tolerance for latency is high.

• Queries are complex or ambiguous.

✅ You're retrieving 20+ candidates

• Large candidate sets benefit most from reranking.

• The improvement justifies the cost.

✅ Your embedding model is underperforming

• Recall@10 <80% with current embeddings.

• Domain-specific queries require specialized reasoning.

Skip the reranker if:

❌ Latency budgets are tight (<200ms target)

• Real-time chat interfaces.

• High-frequency API endpoints.

• User experience is latency-sensitive.

❌ Queries are simple and well-structured

• Factoid queries ("What is X?").

• Navigational queries ("X documentation").

• Bi-encoder already achieves >90% recall.

❌ Small candidate sets (<10 documents)

• Improvement is negligible.

• Better to invest in embedding quality.

Practical implementation patterns

Pattern 1: Hybrid retrieval with selective reranking

Rerank only when necessary:

def retrieve_with_conditional_reranking(query: str, threshold: float = 0.75):
    # Initial retrieval
    candidates = vector_db.search(query, limit=20)
    # Check if top result is confident
    if candidates[0].score > threshold:
        return candidates[:5]  # Skip reranking
    # Rerank if confidence is low
    scores = reranker.predict([[query, c.text] for c in candidates])
    return sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)[:5]

Impact: Reduces reranking overhead by 60% with minimal recall loss.

Pattern 2: Async reranking with progressive disclosure

Show initial results immediately, refine in background:

async def retrieve_with_async_reranking(query: str):
    # Fast initial results
    candidates = await vector_db.search(query, limit=20)
    yield candidates[:5]  # Show immediately
    # Rerank in background
    reranked = await reranker.predict_async(query, candidates)
    yield reranked[:5]  # Update UI with refined results

User experience: Perceived latency reduced by 70%.

Pattern 3: Caching reranker results

Rerank once, cache for similar queries:

from functools import lru_cache
@lru_cache(maxsize=10000)
def rerank_with_cache(query: str, candidates_hash: str):
    return reranker.predict([(query, c.text) for c in candidates])

Impact: 40% reduction in reranking calls for common queries.

Alternative approaches to consider

Before adding a reranker, consider these alternatives:

1. Improve your embedding model

• Fine-tune on domain-specific data.

• Use larger embedding models (e.g., BGE-large instead of BGE-base).

• Switch to hybrid search (dense + sparse).

2. Optimize chunking strategies

• Experiment with chunk sizes (256, 512, 1024 tokens).

• Add overlapping chunks (50-token overlap).

• Use semantic chunking (split on topics, not fixed sizes).

3. Query expansion and reformulation

• Generate multiple query variations.

• Use LLM to rephrase ambiguous queries.

• Extract keywords and entities before retrieval.

4. Ensemble retrieval

• Combine BM25 (lexical) + vector search (semantic).

• Use reciprocal rank fusion (RRF) to merge results.

• Often matches reranker performance with lower latency.

Lessons from production

Extended observation across different deployments reveals consistent patterns:

1. Rerankers are not universal solutions — they help in specific scenarios (ambiguous queries, long documents, precision-critical domains)

2. Latency matters more than expected — users abandon sessions when latency exceeds 300ms, even with better results

3. Cost scales non-linearly — reranking at scale costs 2x more than bi-encoder retrieval alone

4. Fine-tuning pays dividends — domain-specific reranker fine-tuning provides 8-12% recall improvements

5. Hybrid approaches win — selective reranking (only when needed) reduces costs by 60% with minimal quality loss

Conclusion

Rerankers are powerful but overused. They're not "free accuracy" — they trade latency and cost for precision.

Recommended approach:

• Start with a strong bi-encoder embedding model.

• Optimize chunking and retrieval strategies first.

• Add reranking only when precision gaps remain.

• Use selective/async reranking to minimize latency impact.

• Monitor cost and user experience metrics continuously.

The best RAG system isn't the one with every component — it's the one that balances quality, latency, and cost for the specific use case.

---

Working on RAG architecture? Get in touch to discuss system optimization, architecture patterns, and reranker fine-tuning approaches.