What is RAG? The Application Layer of Enterprise AI

Quick answer
Retrieval-augmented generation (RAG) is an AI architecture that improves large language model accuracy by retrieving relevant external information at query time, combining retrieval with generation to produce grounded, citation-backed responses.
Published: April 16, 2026
Last Updated: November 9, 2026
Retrieval-Augmented Generation: Santage Learn cover showing two retrieval paths converging on a single output point, representing retrieved sources combined into one grounded answer

In 2022, the release of ChatGPT made one thing clear to every enterprise in the world: large language models were going to change how work gets done. It also made a second thing clear. These models, for all their fluency, knew nothing about your company, could not access your proprietary data, were frozen at their training cutoff, and occasionally invented facts with confidence. The solution that emerged has a name: retrieval-augmented generation, or RAG.

By 2026, RAG has become one of the standard architectural patterns for enterprise AI and the default way to connect a language model to an organisation's own documents. This guide explains what RAG is, how it works, why it has become essential, and where it is heading.

Key facts about retrieval-augmented generation

What is retrieval-augmented generation?

Retrieval-augmented generation is an AI architecture that improves the accuracy, currency, and groundedness of large language model responses by retrieving relevant external information at the time of a query and using it to generate answers.

Rather than relying solely on the static knowledge encoded during model training, a RAG system actively retrieves relevant documents from a designated knowledge source, passes that retrieved information to the language model as context, and generates responses grounded in the retrieved material.

In short: RAG converts a user's question into a vector, finds the most relevant stored documents using that vector, passes those documents to a language model as context, and generates an answer grounded in the retrieved material.

How does retrieval-augmented generation work?

A RAG system is built from three cooperating components operating in sequence:

How Retrieval-Augmented Generation (RAG) Works: 7-stage pipeline from user query through embedding, vector search, reranking, context injection, LLM generation, to grounded response. Santage.
The 7-stage RAG pipeline. The key innovation is combining retrieval (stages 2-4) with generation (stage 6) to produce grounded, citation-backed responses.
  1. User query: A user submits a question or instruction to the system.
  2. Query embedding: The query is converted into a vector using an embedding model, placing it in the same semantic space as the stored knowledge base.
  3. Vector database search: The query vector is compared against embeddings in a vector database. The system retrieves the most semantically relevant items, typically the top three to ten results.
  4. Reranking and filtering: Retrieved results are passed through a reranking model that orders them by relevance, and through metadata filters for constraints such as date, source authority, or access permissions.
  5. Context injection: The highest-ranked retrieved content is assembled into a prompt along with the original user query.
  6. LLM generation: The language model generates a response using both its pre-trained knowledge and the retrieved context, producing grounded, citation-backed answers.
  7. Response delivery: The generated response is returned to the user, typically with citations pointing back to the retrieved source documents.

The original technique was formalised in the 2020 research paper Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks from researchers at Facebook AI Research, University College London, and New York University.

The core idea behind RAG

RAG separates knowing from reasoning.

In a pure LLM system, knowledge and reasoning are entangled in the model's parameters. You cannot update what the model knows without retraining it. RAG separates these concerns. Reasoning stays in the model. Knowledge moves to a retrievable store. This separation is what makes AI deployable in real business contexts where knowledge changes daily and accuracy is verifiable.

How does RAG differ from related approaches?

ConceptDifference
RAG vs Pure LLMPure LLMs rely on static training knowledge. RAG retrieves current external information at query time
RAG vs Fine-TuningFine-tuning updates model behaviour through retraining. RAG updates knowledge without retraining
RAG vs Semantic SearchSemantic search returns relevant documents. RAG returns a synthesised answer using those documents
RAG vs Traditional SearchTraditional search uses keyword matching. RAG uses semantic similarity plus language generation
RAG vs Prompt EngineeringPrompt engineering refines how you ask. RAG refines what information the model can access
In short: RAG changes what your AI knows. Fine-tuning changes how your AI behaves. The two techniques solve different problems and are usually combined in serious enterprise deployments.

Why is RAG important for modern AI systems?

  1. Solves the knowledge cutoff: RAG connects models to current information, whether that is yesterday's news, this morning's regulatory filing, or your team's latest documentation.
  2. Reduces hallucinations: The original RAG paper found retrieval produced more factual language than a generation-only model (Lewis et al., 2020), and Facebook AI Research showed retrieval substantially reduced hallucinated knowledge in dialogue systems (Shuster et al., 2021).
  3. Enables proprietary data use: Enterprises cannot send sensitive documents to a foundation model for retraining, but they can host those documents in their own vector database and retrieve selectively.
  4. Provides auditable responses: Every RAG answer can be traced back to its source passages, essential for regulated industries.

McKinsey estimated in 2023 that generative AI could add $2.6 trillion to $4.4 trillion a year in value across the 63 use cases it analysed, with about 75% of that value in four areas: customer operations, marketing and sales, software engineering, and research and development (McKinsey, 2023). Most of those use cases depend on a model working accurately with a company's own information, which is the problem RAG solves.

What are the limitations of RAG?

Where is RAG used in practice?

  1. Enterprise knowledge: Microsoft Copilot Studio, Google Workspace with Gemini, Notion AI, and Atlassian Intelligence all use RAG.
  2. Customer support: Zendesk, Intercom, and Salesforce deploy RAG for AI-driven support, grounding answers in help-centre articles and past tickets.
  3. Legal and compliance: Thomson Reuters, LexisNexis, and Harvey AI use RAG for citation-backed legal research.
  4. Healthcare: Clinical decision support and medical literature review increasingly rely on RAG, because answers must cite current guidelines and published evidence rather than a model's training data.
  5. Financial services: Banks and financial data providers use RAG over filings, research, and internal policy documents so that every answer can be traced to its source.
  6. AI search engines: Perplexity, You.com, and AI modes in Google Search and Microsoft Copilot are all RAG systems at their core.
  7. Software development: GitHub Copilot Enterprise and Cursor use RAG to ground code suggestions in an organisation's own codebase.
  8. Scientific research: RAG-enabled tools from Elicit, Scite, and Consensus help researchers synthesise findings across large bodies of literature.

What are the main types of RAG architecture?

RAG is a family of designs, not a single recipe. A widely cited survey groups them into three generations: naive RAG, advanced RAG, and modular RAG (Gao et al., 2023).

Most production systems sit in the advanced RAG category. The more sophisticated designs earn their complexity when questions span many documents, require relationships between entities, or arrive in forms that a single search cannot answer.

What is agentic RAG?

Agentic RAG puts an AI agent in charge of retrieval. In standard RAG, retrieval happens exactly once, before generation, using the user's question as written. In agentic RAG, the model decides whether to search, what to search for, which source to use, and whether the results are good enough, and it can search again based on what it found.

The pattern descends from ReAct, which showed that interleaving reasoning steps with actions such as search queries outperformed reasoning or acting alone on knowledge-intensive tasks (Yao et al., 2022). In practice, retrieval becomes one tool among several. The agent calls a search function through the mechanisms described in our guide to tool use and function calling, and enterprise knowledge sources are increasingly exposed to agents as standard servers through the Model Context Protocol, so the same agent can query a document store, a database, and a ticketing system in one task.

A typical agentic RAG loop works like this:

  1. Plan: The agent breaks a complex question into sub-questions.
  2. Route: For each sub-question, it chooses a source, such as the vector database, a SQL database, the web, or an internal API.
  3. Retrieve and assess: It runs the search and judges whether the results answer the sub-question.
  4. Refine: If not, it reformulates the query or tries another source.
  5. Synthesise: It combines the evidence into a cited answer, or reports what it could not find.

Larger deployments split these roles across specialised agents, one that plans, several that search different sources, and one that checks the final answer, a design covered in our guides to multi-agent systems and AI agent orchestration. A 2025 survey of agentic RAG describes this shift from fixed pipelines to retrieval controlled by autonomous agents as the main direction of the field (Singh et al., 2025).

The trade-offs are real. Agentic RAG answers harder, multi-part questions, but each extra search and reasoning step adds latency and cost, and an agent that can retrieve freely can also retrieve content planted to manipulate it. Many teams use standard RAG for routine questions and escalate to agentic retrieval only when a question needs it.

In short: agentic RAG turns retrieval from a fixed first step into a tool an AI agent uses deliberately, searching, judging, and searching again until it has enough evidence to answer.

How do you evaluate a RAG system?

A RAG system can fail in two places, retrieval and generation, so it has to be measured in both. The RAGAS framework defined three widely used measures that do not require hand-written reference answers: faithfulness, whether every claim in the answer is supported by the retrieved context; answer relevance, whether the answer addresses the question; and context relevance, whether the retrieved passages are focused on what was asked (Es et al., 2023).

Retrieval should also be measured directly, using a labelled set of real questions paired with the documents that should be found. The key metrics are recall, whether the right documents appear in the top results at all, and ranking quality, whether they appear near the top.

Independent testing shows why evaluation matters. A 2024 Stanford study of commercial legal research tools built on RAG found that leading products from LexisNexis and Thomson Reuters still produced incorrect or misgrounded answers between 17% and 33% of the time, despite marketing that suggested otherwise (Magesh et al., 2024). RAG reduces hallucinations; it does not remove the need to measure them.

Evaluation also has to continue after launch. Document collections change, users ask questions the test set never anticipated, and a new embedding model or prompt can quietly lower quality on cases that used to work. Mature teams keep a fixed regression set of real questions with known good answers and rerun it whenever any component changes, sample live answers for automated faithfulness scoring, and route answers with low faithfulness or weak retrieval scores to human review. Logging which passages were retrieved for each answer is what makes this possible: when an answer is wrong, the log shows whether retrieval missed the right document or the model misused a document it was given, and those two failures have different fixes.

Is RAG still needed with long context windows?

Frontier models now accept context windows of hundreds of thousands to millions of tokens, which raises a fair question: why retrieve at all if you can paste in everything?

The research gives a nuanced answer. A Google DeepMind study found that, when given enough resources, long-context models outperformed RAG on average answer quality, but RAG remained far cheaper, and a hybrid that routed each query to RAG or long context matched long-context quality at much lower cost (Li et al., 2024). Long contexts also have a known weakness: models use information at the beginning and end of a long input more reliably than information buried in the middle (Liu et al., 2023).

ApproachBest suited to
Long context aloneA bounded set of documents, such as one contract or one codebase, where cost per query is acceptable
RAGLarge or fast-changing knowledge bases, high query volume, access controls per user, and answers that need citations
RAG plus long contextRetrieve a generous set of relevant material, then let a long-context model reason over all of it
Fine-tuningChanging the model's style, format, or specialised behaviour rather than its knowledge

For enterprise knowledge bases measured in millions of documents, with permissions that differ by user and content that changes daily, retrieval remains necessary. What has changed is that retrieval no longer has to be precise to a few chunks, because the model can now read much more of what is retrieved.

The future of retrieval-augmented generation

  1. Agentic RAG: The simple "embed, retrieve, generate" pattern is evolving into systems that can decide what to retrieve, formulate sub-queries, and iterate. Self-RAG and Corrective RAG research showed that letting the model decide when to retrieve and when to search again improves accuracy over fixed single-pass retrieval (Asai et al., 2023; Yan et al., 2024).
  2. Multimodal RAG: Retrieval is expanding beyond text to images, video, audio, and tables, built on shared embedding spaces such as CLIP that place different data types in one vector space (Radford et al., 2021).
  3. Knowledge graph integration: Hybrid systems combining vector search with knowledge graphs are emerging for enterprise deployments requiring both semantic and structural reasoning. Microsoft Research's GraphRAG showed graph-based retrieval answers broad, corpus-wide questions more comprehensively than standard chunk retrieval (Edge et al., 2024).
  4. Compliance by default: The EU AI Act and sector-specific regulations are pushing RAG toward detailed provenance, citation tracking, and audit trails for every generated claim.

The organisations that build strong RAG capabilities are building a compounding advantage. Every document added to their retrieval corpus makes their AI systems more capable. Every query their agents resolve produces feedback that improves retrieval quality. RAG is not a feature. It is infrastructure, and the organisations that treat it as such will outpace those that do not.

Frequently asked questions

What is the difference between RAG and semantic search?
Semantic search retrieves relevant documents based on meaning. RAG goes further by taking those documents, passing them to a language model, and generating a synthesised answer grounded in the retrieved material.
Does ChatGPT use RAG?
Yes, in several modes. ChatGPT's web browsing feature is a form of RAG. Enterprise ChatGPT deployments use RAG to connect to organisation-specific knowledge.
Why does RAG reduce hallucinations?
RAG grounds the model's response in specific retrieved text. When a model has the exact source material for its answer, it is far less likely to fabricate because it is synthesising rather than generating from memory.
When should you use RAG instead of fine-tuning?
Use RAG for up-to-date, proprietary, or voluminous knowledge. Use fine-tuning to change model behaviour, style, or format. Use both when you need both grounded knowledge and consistent behaviour.
What data sources can RAG systems use?
Virtually any text-based source: documents, PDFs, web pages, databases, APIs, wikis, customer tickets, emails, research papers. Modern multimodal RAG also supports images, audio, and video.
How much does it cost to run a RAG system?
Costs include embedding the documents, hosting the vector index, language model inference for every answer, and optional reranking. Cost scales with document volume, query volume, and the model chosen, and language model inference is usually the largest recurring share. Small deployments can run on minimal budgets.
What is agentic RAG?
Agentic RAG is a form of retrieval-augmented generation in which an AI agent controls retrieval. Instead of searching once before answering, the agent decides what to search for and where, judges the results, and searches again as needed before writing a cited answer.
Is RAG still necessary with long context windows?
Usually yes for large knowledge bases. Research from Google DeepMind found long-context models can match or beat RAG on quality but at much higher cost. RAG also handles fast-changing data, per-user permissions, and citations, and the two approaches are often combined.

Sources and further reading

  1. Lewis, P. et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Facebook AI Research, UCL, NYU, 2020. arxiv.org/abs/2005.11401
  2. McKinsey Global Institute. The economic potential of generative AI: The next productivity frontier. June 2023. mckinsey.com
  3. Shuster, K. et al. Retrieval Augmentation Reduces Hallucination in Conversation. Facebook AI Research, 2021. arxiv.org/abs/2104.07567
  4. Radford, A. et al. Learning Transferable Visual Models From Natural Language Supervision (CLIP). OpenAI, 2021. arxiv.org/abs/2103.00020
  5. European Commission. The EU AI Act. 2025. digital-strategy.ec.europa.eu
  6. World Economic Forum. AI Governance and Future of Work. weforum.org
  7. Gao, Y. et al. Retrieval-Augmented Generation for Large Language Models: A Survey. 2023. arxiv.org/abs/2312.10997
  8. Edge, D. et al. From Local to Global: A Graph RAG Approach to Query-Focused Summarization. Microsoft Research, 2024. arxiv.org/abs/2404.16130
  9. Asai, A. et al. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. 2023. arxiv.org/abs/2310.11511
  10. Yan, S. et al. Corrective Retrieval Augmented Generation. 2024. arxiv.org/abs/2401.15884
  11. Anthropic. Introducing Contextual Retrieval. 2024. anthropic.com/news/contextual-retrieval
  12. Yao, S. et al. ReAct: Synergizing Reasoning and Acting in Language Models. Princeton and Google, 2022. arxiv.org/abs/2210.03629
  13. Singh, A. et al. Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG. 2025. arxiv.org/abs/2501.09136
  14. Es, S. et al. RAGAS: Automated Evaluation of Retrieval Augmented Generation. 2023. arxiv.org/abs/2309.15217
  15. Magesh, V. et al. Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools. Stanford, 2024. arxiv.org/abs/2405.20362
  16. Li, Z. et al. Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach. Google DeepMind, 2024. arxiv.org/abs/2407.16833
  17. Liu, N. F. et al. Lost in the Middle: How Language Models Use Long Contexts. Stanford, 2023. arxiv.org/abs/2307.03172