Published on AskGenAI | June 2026
A field report from building (and breaking) a real-world RAG pipeline — where hybrid retrieval, cross-encoders, and score-gap filtering saved the day, but query rewriting and entity boosting almost destroyed it.
The Setup: What We Built
When we started building a production-grade RAG system for askgenai.in, the architecture looked solid on paper:
| Layer | Technology | Purpose |
|---|---|---|
| Document Ingestion | PyMuPDF + Chunking + PII Redaction | Parse, split, and sanitize PDFs |
| Vector Store | Chroma + MiniLM embeddings | Semantic similarity search |
| Keyword Retrieval | BM25 | Exact and partial keyword matching |
| Fusion | Reciprocal Rank Fusion (RRF) | Combine semantic + keyword rankings |
| Entity Boosting | spaCy NER | Prioritize chunks containing query entities |
| Reranking | Cross-encoder | Reorder fused results by relevance |
| Smart Filtering | Threshold + Score-Gap Logic | Remove noise before LLM generation |
| Query Rewriting | LLM-based selective rewrite | Expand short queries for better retrieval |
| Observability | LangSmith + Cost Tracking | Token usage, latency, and cost logging |
We tested it against a 5-page PDF containing 10 Indian business success stories — Reliance, Tata, Infosys, Myntra, BYJU’S, Paytm, Aadhaar, Nykaa, Ola, and Zoho.
Everything worked in isolation. Then we ran real queries. And that’s when the lessons began.
Lesson 1: Your Semantic Retriever Will Betray You on Named Entities
The Failure:
User asks: “Explain Zoho story”
Semantic retrieval (MMR + MiniLM) returns: Ola, Nandan Nilekani, Myntra
Zoho — the actual subject — is nowhere in the top 3.
Why It Happened:
MiniLM is a general-purpose embedding model. It struggles with proper nouns like “Zoho,” “Nykaa,” and “BYJU’S” because semantic similarity breaks down when the training corpus hasn’t seen these terms in rich contextual relationships. The model latched onto generic business narrative patterns instead of entity-specific meaning.
The Fix (Already in Architecture):
This is exactly why we built hybrid retrieval. BM25 — the keyword retriever — found Zoho immediately because it doesn’t care about semantic meaning; it cares about term frequency and exact matches.
The Insight:
“Without BM25, your RAG pipeline is a coin flip on named entities.”
Your semantic retriever is not “bad” — it’s inconsistent. It handled “Nykaa” fine but choked on “Zoho.” Inconsistency is harder to debug than consistent failure. The only reliable defense is a multi-retriever architecture where keyword search acts as a safety net.
Lesson 2: Query Rewriting Can Turn Into Hallucination Injection
The Failure:
User asks: “Explain Nykaa story”
Our LLM-based rewriter transformed this into:
“Falguni Nayar founded Nykaa in 2012 as an online beauty and wellness retailer, targeting India’s underserved premium beauty market…”
Looks great, right? Wrong.
This isn’t a rewrite — it’s an answer. The LLM hallucinated facts (2012, premium beauty market, customer trust) and injected them into the retrieval query. Now the retriever is searching against generated fiction, not the user’s actual intent.
Why It Still Worked (Dangerously):
The hallucinated query contained enough real keywords (“Nykaa,” “Falguni Nayar”) that retrieval survived. But imagine a query about an unknown startup: the LLM invents facts, retrieval chases ghosts, and the final answer is confidently wrong.
The Fix:
Retrieval query rewriting should be reformulation, not expansion. Rules for production:
- Use only terms present in the original query
- Do not answer the question
- Do not introduce new facts
- Return only a search query
“Explain Nykaa story” → "Nykaa founding story Falguni Nayar journey"
The Insight:
“Your query rewriter is the most dangerous component because it operates before any guardrail. A bad rewrite poisons everything downstream.”
Lesson 3: Entity Boosting Will Boost Garbage If You Let It
The Failure:
For the Nykaa query, our entity detector returned:['the', 'nykaa', 'india', 'she', 'falguni', 'nayar']
Then it boosted 5 chunks because they contained “she” or “the” — words present in virtually every chunk. Entity boosting became uniform noise injection, not targeted relevance amplification.
Why It Happened:
The entity extractor was using raw noun chunks or capitalized tokens without filtering for part-of-speech tags or entity types. Pronouns (“she”) and determiners (“the”) are not entities. Boosting them is mathematically equivalent to boosting nothing — but it adds latency and complexity.
The Fix:
Filter entities by type before boosting. Keep only:
PERSON(Falguni Nayar)ORG(Nykaa, Zoho)PRODUCT(BYJU’S app)GPE(India)
Remove: PRON, DET, STOPWORDS.
The Insight:
“A feature that boosts everything boosts nothing. Entity boosting requires surgical precision, not carpet bombing.”
Lesson 4: RRF Works Exactly As Designed — And That’s the Problem
The Failure (Misdiagnosis):
When we saw RRF scores like 0.032787, 0.016129, 0.016129..., we initially worried the fusion was broken. The top score wasn’t “Zoho” — it was a generic chunk that appeared in both semantic and BM25 rankings.
The Reality:
RRF was working perfectly. It rewards chunks that appear across multiple retrieval systems. The issue wasn’t RRF — it was that both retrieval systems were returning weak results, and RRF dutifully fused weak rankings into a slightly-less-weak ranking.
The Fix:
RRF is not a magic quality improver. It’s a ranking stabilizer. If your individual retrievers are poor, RRF will give you the best of bad options. The real fix is improving the inputs:
- Upgrade embedding model (MiniLM → BGE or E5)
- Tune BM25 parameters (k1, b) for your document length
- Add query expansion synonyms
The Insight:
“RRF doesn’t fix bad retrieval. It just makes bad retrieval democratically bad.”
Lesson 5: The Cross-Encoder Is the True Hero of Your Architecture
The Success:
After all the upstream noise — semantic misses, questionable rewrites, over-boosted entities — the cross-encoder delivered this for the Nykaa query:
| Chunk | Score |
|---|---|
| Nykaa (correct) | 5.8661 |
| Nandan Nilekani | -0.5487 |
| Infosys | -2.2432 |
| Myntra | -2.9138 |
| BYJU’S | -5.9350 |
A 6.4-point gap between correct and second-best. The cross-encoder didn’t just rerank — it obliterated the noise.
Why This Matters:
In production RAG, you need a component that can say: “I don’t care what the retrievers think — this is the right answer.” The cross-encoder is that component. It operates on query-document pairs with full attention, making it far more accurate than bi-encoder retrieval.
The Insight:
“Build your pipeline assuming every upstream component will fail. The cross-encoder is your last line of defense.”
Lesson 6: Score-Gap Filtering Is Aggressive — And That’s Good
The Success:
With a 6.4-point gap between top and second scores, our threshold logic triggered:"Large score gap detected. Returning only top chunk."
Result: The LLM prompt contained only the Nykaa chunk. No Myntra, no BYJU’S, no Infosys noise.
Without This Feature:
The LLM would have received 5 chunks of varying relevance. Even with good instruction following, the model might blend facts, hallucinate connections, or simply consume more tokens for no benefit.
The Insight:
“Sometimes the best context is less context. Score-gap filtering is the art of knowing when to shut up.”
Lesson 7: Observability Is Not Optional — It’s How You Debug Invisible Failures
What We Logged:
Every query generated a trace like this:
Prompt Tokens : 410
Completion Tokens : 141
Total Tokens : 551
Latency (sec) : 1.53
Estimated Cost($) : 0.000692Why This Saved Us:
When semantic retrieval failed on “Zoho,” we didn’t know until we read the logs. When query rewriting hallucinated, we only caught it because the rewritten query was logged verbatim. When entity boosting boosted “the,” we saw the entity list.
Without LangSmith-style tracing, these failures are silent. The pipeline produces an answer, the user is mildly dissatisfied, and you never know why.
The Insight:
“In production RAG, you don’t have a bug until you have a log proving it’s a bug.”
The Architecture That Survived: A Layered Defense Model
Looking back, our pipeline succeeded not because any single component was perfect, but because multiple imperfect components compensated for each other:
User Query
↓
Query Classifier (RAG vs SQL vs Hybrid) ✅
↓
Query Rewriter ⚠️ (sometimes hallucinates)
↓
Semantic Retrieval (MiniLM) ⚠️ (misses named entities)
↓
BM25 Retrieval ✅ (saves semantic failures)
↓
RRF Fusion ✅ (stabilizes rankings)
↓
Entity Boosting ⚠️ (boosts stopwords sometimes)
↓
Cross-Encoder Reranking ⭐⭐⭐⭐⭐ (the real hero)
↓
Threshold + Score-Gap Filtering ⭐⭐⭐⭐⭐ (removes noise)
↓
LLM Generation ✅ (clean prompt, good answer)Any single layer would have failed. Together, they behaved like a production system.
What We Would Do Next
If we paused feature development and focused on scientific improvement:
| Priority | Experiment | Expected Impact |
|---|---|---|
| 1 | Fix entity extraction (remove PRON/DET) | Reduce noise in boosting |
| 2 | Constrain query rewriter (no fact injection) | Prevent hallucination poisoning |
| 3 | Evaluate MiniLM vs BGE vs E5 on named entities | Biggest quality gain |
| 4 | Measure MRR before/after RRF | Quantify fusion value |
| 5 | Add session-state deduplication for ingestion | Reduce redundant compute |
The next major quality improvement won’t come from concurrency or caching. It will come from measuring retrieval quality scientifically and upgrading the embedding model.
Final Thoughts
Building RAG in production is not about implementing the paper-perfect architecture. It’s about building failure-tolerant systems where no single component is trusted completely.
The logs we analyzed weren’t just debugging output — they were a post-mortem of a system that worked despite itself. The semantic retriever failed. The query rewriter hallucinated. The entity booster spammed. And still, the user got the right answer because we built layers.
That’s the unknown side of production RAG. It’s not clean. It’s not elegant. It’s a stack of compensating controls that collectively keep you out of trouble.
Originally published on askgenai.in and Medium