Published on AskGenAI | June 2026
Most RAG benchmarks test AI on public data. This one tests it on the messy reality of enterprise life.
The Problem No One Was Solving
Every company building AI-powered internal search faces the same challenge: how do you know if it actually works?
Existing RAG benchmarks — MS MARCO, Natural Questions, HotpotQA — are built on public data: Wikipedia, web searches, Stack Overflow. They’re clean, well-structured, and nothing like the chaos of actual company knowledge.
Real enterprise data lives in Slack threads, misfiled Google Docs, outdated Confluence pages, email chains with contradictory information, and CRM records that reference projects by codenames no outsider would understand. Until now, there was no public benchmark that captured this reality.
EnterpriseRAG-Bench changes that.
What Is EnterpriseRAG-Bench?
Created by the team at Onyx (a RAG platform company), EnterpriseRAG-Bench is a dataset and evaluation framework designed specifically for company internal documents. It simulates a fictional AI inference company called Redwood Inference and populates it with over 500,000 documents across 9 enterprise sources — then asks 500 carefully crafted questions that test whether a RAG system can actually retrieve and reason over this data.
The Document Corpus
| Source | ~Volume | What It Represents |
|---|---|---|
| Slack | 275,000 | Team chatter, decisions, context lost in threads |
| Gmail | 120,000 | Cross-functional email chains |
| Linear | 35,000 | Engineering and product tickets |
| Google Drive | 25,000 | Collaborative docs, often outdated |
| Hubspot | 15,000 | CRM data, sales records |
| Fireflies | 10,000 | Meeting transcripts |
| GitHub | 8,000 | Code, PRs, technical discussions |
| Jira | 6,000 | Support tickets, bug reports |
| Confluence | 5,000 | Wikis, runbooks, “official” docs |
Notice the distribution: Slack dominates. Confluence is sparse. This mirrors real companies, where institutional knowledge lives in ephemeral chat rather than polished documentation.
The 10 Question Types: A Taxonomy of Enterprise RAG Hardness
The 500 questions aren’t random. They’re organized into 10 categories that map to real failure modes in enterprise RAG:
| Category | Count | The Challenge |
|---|---|---|
| Basic | 175 | Simple lookup — one document, obvious answer. The baseline. |
| Semantic | 125 | Same difficulty, but no keyword overlap. Tests if your embeddings actually understand meaning. |
| Intra-Document Reasoning | 40 | Answer requires connecting facts from sections far apart in one long document. |
| Project Related | 40 | Aggregate scattered updates across multiple docs from one initiative. |
| Constrained | 30 | Multiple docs look relevant, but qualifiers (dates, roles, versions) eliminate all but one. |
| Conflicting Info | 20 | Documents directly contradict each other. Can your system detect and resolve this? |
| Completeness | 20 | You need all relevant documents (up to 10) — partial retrieval gives a wrong answer. |
| Miscellaneous | 20 | Informal, off-topic, or loosely organized content. The “junk drawer” test. |
| High Level | 10 | No single document contains the answer. Requires synthesis. |
| Info Not Found | 20 | The answer isn’t in the corpus. Tests hallucination resistance. |
This taxonomy is the real contribution. It gives RAG engineers a diagnostic tool: if your system fails on “Conflicting Info” questions, you know you need better de-duplication or contradiction detection. If it fails on “Semantic” questions, your embedding model is too keyword-dependent.
What Makes the Data “Real”
The Onyx team didn’t just generate documents and call it a day. They followed five design principles that distinguish this from synthetic benchmarks:
- Cross-document coherence — Redwood Inference has a real org chart, product roadmap, and project codenames. Documents reference each other meaningfully.
- Realistic volume distribution — Slack overwhelms wikis, just like in your company.
- Realistic noise — Misfiled documents, outdated versions, near-duplicates with updated facts. The dataset is intentionally messy.
- Internal terminology — Acronyms and project names that are meaningless without company context.
- Generality — The generation framework is adaptable to other industries and company sizes.
How to Use It
The dataset is available via:
- GitHub Releases —
all_documents.zipor sliced by source type - HuggingFace — for easy integration into training pipelines
- Questions — in
questions.jsonlwith ground truth document IDs
There’s also a leaderboard open for submissions (Onyx excludes itself to avoid conflict of interest). Submissions require reproducibility — either open-source code or a sandbox endpoint for verification.
Why This Matters for the RAG Ecosystem
EnterpriseRAG-Bench arrives at a critical moment. Companies are rushing to deploy internal AI assistants, but most are flying blind on evaluation. This benchmark gives them:
- A realistic testbed that mirrors their actual data landscape
- Granular diagnostics through the 10 question categories
- A generation framework to create custom benchmarks for their specific industry
If you’re building RAG for internal use, this isn’t just another leaderboard to chase. It’s a reality check.
The Paper & Citation
The research paper is available on arXiv:
@misc{sun2026enterpriseragbench,
title={EnterpriseRAG-Bench: A RAG Benchmark for Company Internal Knowledge},
author={Sun, Yuhong and Rahmfeld, Joachim and Weaver, Chris and Desai, Roshan and Huang, Wenxi and Butler, Mark H.},
year={2026},
eprint={2605.05253},
archivePrefix={arXiv},
primaryClass={cs.IR}
}Bottom Line
EnterpriseRAG-Bench is the first benchmark that takes enterprise RAG seriously. It doesn’t pretend company knowledge is clean, structured, or public. It embraces the chaos — and gives us a way to measure whether our AI systems can navigate it.
If your RAG pipeline aces MS MARCO but fails in production, this is probably why. And now, finally, there’s a benchmark that will show you exactly where.
Want to explore the dataset or submit to the leaderboard? Head to the GitHub repository.