In 2022, the release of ChatGPT made one thing clear to every enterprise in the world: large language models were going to change how work gets done. It also made a second thing clear. These models, for all their fluency, knew nothing about your company, could not access your proprietary data, were frozen at their training cutoff, and occasionally invented facts with confidence. The solution that emerged has a name: retrieval-augmented generation, or RAG.
By 2026, RAG has become one of the standard architectural patterns for enterprise AI and the default way to connect a language model to an organisation's own documents. This guide explains what RAG is, how it works, why it has become essential, and where it is heading.
- What it is: An AI architecture combining information retrieval with language model generation to produce grounded, accurate responses
- Why it matters: Solves the knowledge cutoff problem, reduces hallucinations, enables safe use of proprietary data, and provides auditable responses
- Known limit: A 2024 Stanford study found leading RAG-based legal research tools still produced incorrect or misgrounded answers 17% to 33% of the time
- First introduced: 2020 research paper by Facebook AI Research, UCL, and NYU
- Core components: Embedding model, vector database, optional reranker, and large language model
- Primary use cases: Enterprise knowledge management, customer support, legal research, AI search engines, clinical decision support
- Common frameworks: LangChain and LlamaIndex are widely used open-source RAG toolkits
What is retrieval-augmented generation?
Retrieval-augmented generation is an AI architecture that improves the accuracy, currency, and groundedness of large language model responses by retrieving relevant external information at the time of a query and using it to generate answers.
Rather than relying solely on the static knowledge encoded during model training, a RAG system actively retrieves relevant documents from a designated knowledge source, passes that retrieved information to the language model as context, and generates responses grounded in the retrieved material.
How does retrieval-augmented generation work?
A RAG system is built from three cooperating components operating in sequence:
- User query: A user submits a question or instruction to the system.
- Query embedding: The query is converted into a vector using an embedding model, placing it in the same semantic space as the stored knowledge base.
- Vector database search: The query vector is compared against embeddings in a vector database. The system retrieves the most semantically relevant items, typically the top three to ten results.
- Reranking and filtering: Retrieved results are passed through a reranking model that orders them by relevance, and through metadata filters for constraints such as date, source authority, or access permissions.
- Context injection: The highest-ranked retrieved content is assembled into a prompt along with the original user query.
- LLM generation: The language model generates a response using both its pre-trained knowledge and the retrieved context, producing grounded, citation-backed answers.
- Response delivery: The generated response is returned to the user, typically with citations pointing back to the retrieved source documents.
The original technique was formalised in the 2020 research paper Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks from researchers at Facebook AI Research, University College London, and New York University.
The core idea behind RAG
RAG separates knowing from reasoning.
In a pure LLM system, knowledge and reasoning are entangled in the model's parameters. You cannot update what the model knows without retraining it. RAG separates these concerns. Reasoning stays in the model. Knowledge moves to a retrievable store. This separation is what makes AI deployable in real business contexts where knowledge changes daily and accuracy is verifiable.
How does RAG differ from related approaches?
| Concept | Difference |
|---|---|
| RAG vs Pure LLM | Pure LLMs rely on static training knowledge. RAG retrieves current external information at query time |
| RAG vs Fine-Tuning | Fine-tuning updates model behaviour through retraining. RAG updates knowledge without retraining |
| RAG vs Semantic Search | Semantic search returns relevant documents. RAG returns a synthesised answer using those documents |
| RAG vs Traditional Search | Traditional search uses keyword matching. RAG uses semantic similarity plus language generation |
| RAG vs Prompt Engineering | Prompt engineering refines how you ask. RAG refines what information the model can access |
Why is RAG important for modern AI systems?
- Solves the knowledge cutoff: RAG connects models to current information, whether that is yesterday's news, this morning's regulatory filing, or your team's latest documentation.
- Reduces hallucinations: The original RAG paper found retrieval produced more factual language than a generation-only model (Lewis et al., 2020), and Facebook AI Research showed retrieval substantially reduced hallucinated knowledge in dialogue systems (Shuster et al., 2021).
- Enables proprietary data use: Enterprises cannot send sensitive documents to a foundation model for retraining, but they can host those documents in their own vector database and retrieve selectively.
- Provides auditable responses: Every RAG answer can be traced back to its source passages, essential for regulated industries.
McKinsey estimated in 2023 that generative AI could add $2.6 trillion to $4.4 trillion a year in value across the 63 use cases it analysed, with about 75% of that value in four areas: customer operations, marketing and sales, software engineering, and research and development (McKinsey, 2023). Most of those use cases depend on a model working accurately with a company's own information, which is the problem RAG solves.
What are the limitations of RAG?
- Retrieval quality determines output quality: If the retrieval layer returns irrelevant documents, the LLM generates responses grounded in wrong information.
- Latency accumulation: Every RAG query involves embedding, vector search, reranking, and LLM generation. These steps add latency.
- Chunking strategy matters: How documents are split into retrievable chunks significantly affects retrieval quality. This remains one of the most underappreciated engineering challenges.
- Context window constraints: Retrieving too much information can degrade model performance due to the "lost in the middle" phenomenon.
- Hallucination reduction, not elimination: RAG significantly reduces hallucinations but does not eliminate them. Models can still misinterpret retrieved information.
- Maintenance burden: A production RAG system has more moving parts than a pure LLM deployment. Embedding models, indexes, and document sources must all be maintained.
Where is RAG used in practice?
- Enterprise knowledge: Microsoft Copilot Studio, Google Workspace with Gemini, Notion AI, and Atlassian Intelligence all use RAG.
- Customer support: Zendesk, Intercom, and Salesforce deploy RAG for AI-driven support, grounding answers in help-centre articles and past tickets.
- Legal and compliance: Thomson Reuters, LexisNexis, and Harvey AI use RAG for citation-backed legal research.
- Healthcare: Clinical decision support and medical literature review increasingly rely on RAG, because answers must cite current guidelines and published evidence rather than a model's training data.
- Financial services: Banks and financial data providers use RAG over filings, research, and internal policy documents so that every answer can be traced to its source.
- AI search engines: Perplexity, You.com, and AI modes in Google Search and Microsoft Copilot are all RAG systems at their core.
- Software development: GitHub Copilot Enterprise and Cursor use RAG to ground code suggestions in an organisation's own codebase.
- Scientific research: RAG-enabled tools from Elicit, Scite, and Consensus help researchers synthesise findings across large bodies of literature.
What are the main types of RAG architecture?
RAG is a family of designs, not a single recipe. A widely cited survey groups them into three generations: naive RAG, advanced RAG, and modular RAG (Gao et al., 2023).
- Naive RAG: The basic pipeline of embed, retrieve the top few chunks, and generate. It is quick to build and adequate for simple question answering over a clean document set.
- Advanced RAG: Adds steps before and after retrieval. Before retrieval, the system may rewrite or expand the query. During retrieval, it may run hybrid search that combines embedding similarity with keyword matching. After retrieval, a reranking model reorders candidates. Anthropic reported that adding short context to each chunk before embedding cut failed retrievals by 49%, and by 67% when combined with reranking (Anthropic, 2024).
- Modular RAG: Treats retrieval, routing, memory, and generation as interchangeable components that can be arranged differently for different queries.
- GraphRAG: Builds a knowledge graph from the source documents and summarises clusters of related entities in advance. Microsoft Research found it substantially improved the comprehensiveness and diversity of answers to broad questions about an entire corpus, such as "what are the main themes in these documents?", which standard chunk retrieval handles poorly (Edge et al., 2024).
- Self-correcting RAG: Lets the model judge its own retrieval. Self-RAG trains the model to decide when to retrieve and to critique whether retrieved passages support its answer (Asai et al., 2023), and Corrective RAG adds an evaluator that triggers a fresh search when retrieved documents look unreliable (Yan et al., 2024).
Most production systems sit in the advanced RAG category. The more sophisticated designs earn their complexity when questions span many documents, require relationships between entities, or arrive in forms that a single search cannot answer.
What is agentic RAG?
Agentic RAG puts an AI agent in charge of retrieval. In standard RAG, retrieval happens exactly once, before generation, using the user's question as written. In agentic RAG, the model decides whether to search, what to search for, which source to use, and whether the results are good enough, and it can search again based on what it found.
The pattern descends from ReAct, which showed that interleaving reasoning steps with actions such as search queries outperformed reasoning or acting alone on knowledge-intensive tasks (Yao et al., 2022). In practice, retrieval becomes one tool among several. The agent calls a search function through the mechanisms described in our guide to tool use and function calling, and enterprise knowledge sources are increasingly exposed to agents as standard servers through the Model Context Protocol, so the same agent can query a document store, a database, and a ticketing system in one task.
A typical agentic RAG loop works like this:
- Plan: The agent breaks a complex question into sub-questions.
- Route: For each sub-question, it chooses a source, such as the vector database, a SQL database, the web, or an internal API.
- Retrieve and assess: It runs the search and judges whether the results answer the sub-question.
- Refine: If not, it reformulates the query or tries another source.
- Synthesise: It combines the evidence into a cited answer, or reports what it could not find.
Larger deployments split these roles across specialised agents, one that plans, several that search different sources, and one that checks the final answer, a design covered in our guides to multi-agent systems and AI agent orchestration. A 2025 survey of agentic RAG describes this shift from fixed pipelines to retrieval controlled by autonomous agents as the main direction of the field (Singh et al., 2025).
The trade-offs are real. Agentic RAG answers harder, multi-part questions, but each extra search and reasoning step adds latency and cost, and an agent that can retrieve freely can also retrieve content planted to manipulate it. Many teams use standard RAG for routine questions and escalate to agentic retrieval only when a question needs it.
How do you evaluate a RAG system?
A RAG system can fail in two places, retrieval and generation, so it has to be measured in both. The RAGAS framework defined three widely used measures that do not require hand-written reference answers: faithfulness, whether every claim in the answer is supported by the retrieved context; answer relevance, whether the answer addresses the question; and context relevance, whether the retrieved passages are focused on what was asked (Es et al., 2023).
Retrieval should also be measured directly, using a labelled set of real questions paired with the documents that should be found. The key metrics are recall, whether the right documents appear in the top results at all, and ranking quality, whether they appear near the top.
Independent testing shows why evaluation matters. A 2024 Stanford study of commercial legal research tools built on RAG found that leading products from LexisNexis and Thomson Reuters still produced incorrect or misgrounded answers between 17% and 33% of the time, despite marketing that suggested otherwise (Magesh et al., 2024). RAG reduces hallucinations; it does not remove the need to measure them.
Evaluation also has to continue after launch. Document collections change, users ask questions the test set never anticipated, and a new embedding model or prompt can quietly lower quality on cases that used to work. Mature teams keep a fixed regression set of real questions with known good answers and rerun it whenever any component changes, sample live answers for automated faithfulness scoring, and route answers with low faithfulness or weak retrieval scores to human review. Logging which passages were retrieved for each answer is what makes this possible: when an answer is wrong, the log shows whether retrieval missed the right document or the model misused a document it was given, and those two failures have different fixes.
Is RAG still needed with long context windows?
Frontier models now accept context windows of hundreds of thousands to millions of tokens, which raises a fair question: why retrieve at all if you can paste in everything?
The research gives a nuanced answer. A Google DeepMind study found that, when given enough resources, long-context models outperformed RAG on average answer quality, but RAG remained far cheaper, and a hybrid that routed each query to RAG or long context matched long-context quality at much lower cost (Li et al., 2024). Long contexts also have a known weakness: models use information at the beginning and end of a long input more reliably than information buried in the middle (Liu et al., 2023).
| Approach | Best suited to |
|---|---|
| Long context alone | A bounded set of documents, such as one contract or one codebase, where cost per query is acceptable |
| RAG | Large or fast-changing knowledge bases, high query volume, access controls per user, and answers that need citations |
| RAG plus long context | Retrieve a generous set of relevant material, then let a long-context model reason over all of it |
| Fine-tuning | Changing the model's style, format, or specialised behaviour rather than its knowledge |
For enterprise knowledge bases measured in millions of documents, with permissions that differ by user and content that changes daily, retrieval remains necessary. What has changed is that retrieval no longer has to be precise to a few chunks, because the model can now read much more of what is retrieved.
The future of retrieval-augmented generation
- Agentic RAG: The simple "embed, retrieve, generate" pattern is evolving into systems that can decide what to retrieve, formulate sub-queries, and iterate. Self-RAG and Corrective RAG research showed that letting the model decide when to retrieve and when to search again improves accuracy over fixed single-pass retrieval (Asai et al., 2023; Yan et al., 2024).
- Multimodal RAG: Retrieval is expanding beyond text to images, video, audio, and tables, built on shared embedding spaces such as CLIP that place different data types in one vector space (Radford et al., 2021).
- Knowledge graph integration: Hybrid systems combining vector search with knowledge graphs are emerging for enterprise deployments requiring both semantic and structural reasoning. Microsoft Research's GraphRAG showed graph-based retrieval answers broad, corpus-wide questions more comprehensively than standard chunk retrieval (Edge et al., 2024).
- Compliance by default: The EU AI Act and sector-specific regulations are pushing RAG toward detailed provenance, citation tracking, and audit trails for every generated claim.
The organisations that build strong RAG capabilities are building a compounding advantage. Every document added to their retrieval corpus makes their AI systems more capable. Every query their agents resolve produces feedback that improves retrieval quality. RAG is not a feature. It is infrastructure, and the organisations that treat it as such will outpace those that do not.
Frequently asked questions
Sources and further reading
- Lewis, P. et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Facebook AI Research, UCL, NYU, 2020. arxiv.org/abs/2005.11401
- McKinsey Global Institute. The economic potential of generative AI: The next productivity frontier. June 2023. mckinsey.com
- Shuster, K. et al. Retrieval Augmentation Reduces Hallucination in Conversation. Facebook AI Research, 2021. arxiv.org/abs/2104.07567
- Radford, A. et al. Learning Transferable Visual Models From Natural Language Supervision (CLIP). OpenAI, 2021. arxiv.org/abs/2103.00020
- European Commission. The EU AI Act. 2025. digital-strategy.ec.europa.eu
- World Economic Forum. AI Governance and Future of Work. weforum.org
- Gao, Y. et al. Retrieval-Augmented Generation for Large Language Models: A Survey. 2023. arxiv.org/abs/2312.10997
- Edge, D. et al. From Local to Global: A Graph RAG Approach to Query-Focused Summarization. Microsoft Research, 2024. arxiv.org/abs/2404.16130
- Asai, A. et al. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. 2023. arxiv.org/abs/2310.11511
- Yan, S. et al. Corrective Retrieval Augmented Generation. 2024. arxiv.org/abs/2401.15884
- Anthropic. Introducing Contextual Retrieval. 2024. anthropic.com/news/contextual-retrieval
- Yao, S. et al. ReAct: Synergizing Reasoning and Acting in Language Models. Princeton and Google, 2022. arxiv.org/abs/2210.03629
- Singh, A. et al. Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG. 2025. arxiv.org/abs/2501.09136
- Es, S. et al. RAGAS: Automated Evaluation of Retrieval Augmented Generation. 2023. arxiv.org/abs/2309.15217
- Magesh, V. et al. Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools. Stanford, 2024. arxiv.org/abs/2405.20362
- Li, Z. et al. Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach. Google DeepMind, 2024. arxiv.org/abs/2407.16833
- Liu, N. F. et al. Lost in the Middle: How Language Models Use Long Contexts. Stanford, 2023. arxiv.org/abs/2307.03172
