In production RAG, candidates are asked why we don’t just rely on vector similarity search. Why add a second reranking stage? What problem does it solve? When is it worth the latency cost?
Vector search (first stage) is fast but coarse — it retrieves semantically similar chunks using approximate nearest neighbors (ANN), which can miss fine-grained relevance signals. A cross-encoder reranker (second stage) performs full attention over query+document pairs, giving precise relevance scores. Use two-stage when: (1) top-k from vector search has low precision, (2) you can afford 50–200ms extra latency, (3) your corpus is large enough that ANN recall matters. For speed-critical apps, use ColBERT or late-interaction models instead of full cross-encoders.