How do you retrieve for diversity and coverage in RAG instead of returning the top-k similar chunks

In production RAG systems, retrieving the top-k most similar chunks is often insufficient because semantic similarity tends to return duplicate information.

The objective should be maximizing both relevance and coverage.

My approach would be:

1. Maximum Marginal Relevance (MMR)

Instead of selecting chunks purely based on similarity to the query, MMR balances:

– Query relevance

– Diversity among retrieved chunks

This prevents retrieval of five nearly identical passages.

2. Hybrid Search

I combine:

– Dense retrieval (embeddings)

– Sparse retrieval (BM25/keyword search)

This improves recall and ensures important keyword matches are not missed.

3. Metadata Diversification

I enforce diversity across:

– Document sources

– Authors

– Time periods

– Categories

This prevents a single document from dominating retrieval.

4. Query Decomposition

Complex questions are broken into sub-questions.

Example:

“Explain RAG cost optimization.”

Can become:

– Embedding optimization

– Vector database optimization

– Retrieval optimization

– LLM inference optimization

Each sub-query retrieves its own evidence.

5. Cross-Encoder Reranking

Retrieve top 50 candidates.

Use a reranker model to identify the most useful and contextually relevant passages.

6. Parent-Child Retrieval

Store smaller chunks for search.

Return larger parent sections for generation.

This improves context quality while maintaining retrieval precision.

7. Evaluation

I monitor:

– Context Precision

– Context Recall

– Answer Relevance

– Faithfulness

– Source Coverage

using LangSmith, Ragas, and custom evaluation datasets.

Production Strategy

My preferred architecture is:

Query

→ Hybrid Retrieval

→ MMR Diversification

→ Cross-Encoder Reranking

→ Parent Retrieval

→ LLM Generation

This provides better coverage than simple top-k retrieval while maintaining low latency and high answer quality.