How would you route queries between a cheap semantic search and an expensive reranker? Design latency budgets and the routing logic.
Askgenai In Answered question
Architecture:
plain
<code class="language-plain">User Query → Embedding Model → ANN Search (Top-100) → [Router] → Reranker (Top-5) → LLM
↓
Fast Path (skip reranker)</code>
Routing Logic:
-
Query complexity classifier: A lightweight model (DistilBERT, ~50MB) classifies queries as:
- Simple: Factual lookups, single-entity queries → skip reranker, return Top-3 from ANN
- Complex: Multi-hop reasoning, comparative questions → full reranker pipeline
- Confidence-based routing: If the ANN top-1 score is >0.85 cosine similarity, skip reranker. This catches ~40% of queries in practice.
- User tier routing: Free-tier users get fast path only; paid tiers get full reranker.
Latency Budgets (P95):
Table
Stage
Budget
Notes
Embedding
50ms
Batch queries when possible
ANN Search
30ms
HNSW with ef=128
Reranker (ColBERT)
100ms
Optional; only for complex queries
LLM Generation
500ms
Streaming response starts at TTFB
Total
<200ms fast path / <700ms full path
Cost Math:
- ANN search: ~$0.0001/query
- ColBERT reranker: ~$0.001/query (10× more expensive)
- By routing 60% to fast path, you cut reranker costs by 60% with minimal accuracy loss.
Askgenai In Answered question