In the architecture of modern AI, three layers sit in a stack. Large language models provide reasoning. Embeddings translate meaning into math. And between them, holding the entire edifice together, sits a layer of infrastructure that did not meaningfully exist five years ago: the vector database.
Vector databases are the reason AI systems can search the meaning of a billion documents in milliseconds. They are the infrastructure that makes retrieval-augmented generation possible at enterprise scale. And they have moved from niche to standard in a few years: vector search is now built into PostgreSQL, Elasticsearch, MongoDB, and the major cloud data platforms, alongside a field of dedicated vector databases.
- What they are: Specialised data systems that index and search high-dimensional vectors using similarity-based queries rather than exact matches
- Why they matter: The infrastructure layer that makes retrieval-augmented generation and enterprise AI search possible at production scale
- Leading systems: Pinecone, Weaviate, Milvus, Qdrant, Chroma, and pgvector for PostgreSQL-based deployments
- Core algorithm: Most production systems use HNSW (Hierarchical Navigable Small World) for approximate nearest neighbour search
- Typical performance: Millisecond-scale queries; Microsoft Research's DiskANN searched one billion vectors at under 3 ms mean latency on a single machine
- Primary use cases: Retrieval-augmented generation, semantic search, recommendation systems, anomaly detection, AI agent memory
- Market position: Offered both as dedicated systems and as built-in features of PostgreSQL, Elasticsearch, MongoDB Atlas, and the major cloud data platforms
What is a vector database?
A vector database is a specialised data storage and retrieval system designed to index and search high-dimensional vectors, typically embeddings produced by AI models, using similarity-based queries rather than exact matches.
Unlike a traditional relational database that answers "find all customers named Smith," a vector database answers "find the ten documents most similar in meaning to this query." It compares the mathematical distance between stored vectors and a query vector to return the closest matches.
How do vector databases work?
The architecture of a vector database is purpose-built for one task: finding the nearest neighbours in high-dimensional space, as fast as possible.
- Ingestion: Raw data is passed through an embedding model that converts it into vectors. These vectors, along with metadata, are stored in the database.
- Indexing: The database builds specialised index structures for high-dimensional similarity search. The most common is HNSW (Hierarchical Navigable Small World), a graph-based algorithm.
- Query embedding: When a user query arrives, it is embedded using the same model, ensuring the query vector exists in the same semantic space as stored vectors.
- Approximate nearest neighbour search: The database searches the index to find the closest vectors. ANN algorithms trade a small amount of accuracy for enormous gains in speed.
- Metadata filtering and hybrid search: Most production queries combine vector similarity with traditional filters and keyword matching (BM25) in a pattern called hybrid search.
- Ranked results: The database returns the top matches, ranked by similarity score, typically within milliseconds.
The core idea behind vector databases
Vector databases search by meaning, not by match.
A traditional database asks "does this record exactly equal the query?" A vector database asks "which stored records are most similar in meaning to the query?" The first is deterministic and binary. The second is probabilistic and continuous.
How do vector databases differ from related systems?
| Concept | Difference |
|---|---|
| Vector DB vs Relational DB | Relational databases use exact matches on structured data. Vector databases use similarity search on high-dimensional vectors |
| Vector DB vs Search Engine | Search engines excel at keyword matching. Vector databases excel at semantic similarity. Modern hybrid systems combine both |
| Vector DB vs Vector Library | Libraries like FAISS provide algorithms but lack database features such as persistence, replication, and concurrent access |
| Vector DB vs Vector Index | A vector index is a data structure. A vector database is a full system managing indexes, data, metadata, and queries |
| Vector DB vs Graph DB | Graph databases excel at relationship traversal. Vector databases excel at similarity across unstructured data |
How do vector databases power RAG systems?
Retrieval-augmented generation is the single most important application pattern for vector databases. In a RAG system, the vector database holds embeddings of an organisation's knowledge base. When a user asks a question, the system retrieves the most relevant documents and passes them to a large language model as context.
The pattern traces to the 2020 RAG paper from Facebook AI Research, University College London, and New York University, which paired a neural retriever over a dense vector index of Wikipedia with a text generator and found it produced more specific, diverse, and factual language than a generation-only model (Lewis et al., 2020). It is now the default way to connect a language model to private knowledge.
What are the leading vector databases in 2026?
- Pinecone: An early fully managed vector database. Prioritises ease of use and serverless operations.
- Weaviate: Open-source with enterprise support. Distinctive strength in native hybrid search that combines vector and keyword retrieval.
- Milvus: The scalability leader. Engineered for billions-of-vectors workloads with GPU acceleration and distributed deployment. Maintained by Zilliz.
- Qdrant: Performance-focused open-source option. Written in Rust. Strong support for complex metadata filtering.
- Chroma: Developer-experience-first option for rapid prototyping with minimal configuration.
- pgvector: PostgreSQL extension adding vector search to the most widely deployed relational database in the world.
What are the challenges of using vector databases?
- Infrastructure complexity: High-performance vector databases require careful capacity planning, index tuning, and memory management.
- Embedding model coupling: Changing embedding models typically requires re-embedding and re-indexing entire corpora.
- Cost at scale: A single 1,536-dimensional vector requires approximately 6 KB of storage. A billion vectors requires 6 TB before indexes.
- Data freshness: High-performance indexes often require batch updates rather than real-time inserts.
- Regulatory considerations: Embeddings can be partially inverted back into the original text (Morris et al., 2023), so a vector store built from personal data should be governed as personal data under GDPR and the data governance requirements of the EU AI Act.
Where are vector databases used in practice?
- Enterprise knowledge: Microsoft Copilot, Google Workspace with Gemini, Notion AI, and Atlassian Intelligence all rely on vector databases.
- Customer support: Zendesk, Intercom, and Salesforce use vector databases for AI-driven support.
- AI search engines: Perplexity, You.com, and newer AI-first search engines use vector databases as core retrieval infrastructure.
- Financial services: Banks and asset managers use vector search for document analysis, compliance review, and research retrieval across filings and internal reports.
- Healthcare: Mayo Clinic and pharmaceutical companies use vector databases to search clinical literature and match patients to trials.
- Cybersecurity: CrowdStrike and Palo Alto Networks use vector databases to detect anomalous patterns across networks.
- AI agent memory: Emerging agent frameworks use vector databases as the memory layer for long-running autonomous agents.
What indexing algorithms do vector databases use?
Comparing a query against every stored vector, called exact or brute-force search, is accurate but scales poorly: a billion vectors means a billion distance calculations per query. Vector databases instead use approximate nearest neighbour (ANN) indexes that check a small fraction of the data and accept a small chance of missing the true closest match. Four families of index dominate.
- HNSW (Hierarchical Navigable Small World): A layered graph in which each vector links to its near neighbours. A search starts at a sparse top layer, jumps across long distances, then descends to denser layers to refine. Its authors showed search cost growing roughly logarithmically with dataset size (Malkov and Yashunin, 2016). HNSW is fast and accurate, which is why it is the default in most vector databases, but it keeps vectors and graph links in memory, which makes it expensive at very large scale.
- IVF (inverted file index): Vectors are grouped into clusters around central points. A query is compared with the cluster centres first, and only the vectors in the nearest few clusters are searched. IVF uses less memory than HNSW and builds faster, at some cost in recall.
- Product quantization (PQ): Each vector is split into segments, and each segment is replaced by a short code from a learned codebook, compressing vectors many times over (Jégou, Douze, and Schmid, 2011). PQ is usually combined with IVF so that billions of vectors fit in memory.
- Disk-based graphs: Microsoft Research's DiskANN keeps the full index on solid-state storage with only compressed data in memory. Its authors searched a billion-point dataset on a single workstation with 64 GB of RAM at under 3 milliseconds mean latency and over 95% recall (Subramanya et al., NeurIPS 2019).
Every index trades among three quantities: recall, meaning the share of true nearest neighbours found; latency; and memory or cost. Index parameters let operators move along that curve. Meta's open-source FAISS library, used inside many commercial products, implements most of these methods and made GPU-accelerated billion-scale search practical (Johnson, Douze, and Jégou, 2017).
Do you need a dedicated vector database?
Not always. Since 2023, vector search has become a feature of databases many organisations already run. PostgreSQL offers it through the pgvector extension, and MongoDB Atlas, Elasticsearch, OpenSearch, Redis, and the major cloud data warehouses all provide vector indexes. For many teams, the right first step is to use the database they already operate.
| Situation | Usually the better fit |
|---|---|
| Up to a few million vectors, data already in a relational database | Vector extension of the existing database, such as pgvector |
| Search must combine vectors with complex joins, transactions, or permissions | Existing operational database with vector support |
| Hundreds of millions to billions of vectors with strict latency targets | Dedicated vector database built for distributed ANN search |
| Heavy real-time ingestion alongside high query volume | Dedicated vector database or managed service |
| Prototype or single-developer project | Lightweight embedded option or managed free tier |
The deciding factors are scale, latency requirements, how often data changes, and operational capacity. A dedicated system adds another component to secure, monitor, and keep in sync with the source of truth.
Synchronisation is the cost teams most often underestimate. When a document is edited, deleted, or has its access permissions changed in the system of record, the matching vectors must be updated or removed too, or the AI system will keep retrieving content that no longer exists or that a user should no longer see. Keeping vectors inside the same database as the source data makes this consistency automatic. Keeping them in a separate system requires a reliable pipeline that propagates every change, plus monitoring to catch the cases where it fails. For many organisations, that operational burden, rather than raw search speed, is what decides the architecture. A practical pattern is to start with vector search inside the existing database, measure recall and latency on real traffic, and move to a dedicated system only when those measurements show the need.
How do vector databases serve as memory for AI agents?
The fastest-growing use of vector databases in 2026 is as the memory layer for AI agents. A large language model on its own remembers nothing between sessions and can only hold a limited amount of text in its context window. Agents that run for hours, serve the same user over weeks, or coordinate with other agents need somewhere durable to store what they learn.
Research systems established the pattern. Stanford and Google's "generative agents" kept a memory stream of observations and retrieved entries by a combination of recency, importance, and embedding similarity (Park et al., 2023). MemGPT treated the context window like a computer's main memory and a vector store like its disk, moving information between them as needed (Packer et al., 2023).
In production, agent memory typically has three tiers: the current context window, a vector store of past interactions and learned facts, and structured records in conventional databases. The vector store is often exposed to the agent as a tool it can query, following the patterns described in our guide to tool use and function calling, and increasingly through a standard server interface such as the Model Context Protocol. In multi-agent systems, a shared vector store lets specialised agents build on each other's findings, which makes the store's partitioning and access rules part of the orchestration design.
Memory also creates a new attack surface. Anything an agent retrieves enters its context and can influence its actions. Researchers demonstrated indirect prompt injection, in which instructions hidden in retrieved content take control of the model (Greshake et al., 2023). The PoisonedRAG study found that inserting five crafted texts per target question into a knowledge base of millions of documents achieved about a 90% attack success rate (Zou et al., 2024). For agents that can take real actions, write access to the vector store must be controlled as carefully as write access to code.
Agent memory also needs housekeeping that document search rarely does. Memories should be isolated per user or tenant so one customer's history never surfaces in another's session, deduplicated so repeated observations do not crowd out new ones, and given expiry or review rules so outdated facts do not keep steering decisions. Each stored entry should record its source and time, so that when an agent acts on a memory, an operator can trace where that belief came from.
How do you evaluate a vector database?
Vendor benchmarks are hard to compare because each tunes its own system for the test. Independent harnesses such as ANN-Benchmarks measure recall against queries per second across algorithms on standard datasets (Aumüller, Bernhardsson, and Faithfull, 2018), but the only reliable test is a workload that resembles yours. The metrics that matter are:
- Recall at k: The share of true top-k neighbours returned. Measure it against brute-force results on a sample of real queries.
- Tail latency: The 95th and 99th percentile response times under realistic load, not the average.
- Filtered performance: Speed and recall when queries include metadata filters such as user, date, or permission. Many systems slow down or lose recall sharply under restrictive filters.
- Ingestion and freshness: How quickly new or updated vectors become searchable, and whether heavy writes degrade query speed.
- Total cost: Memory, storage, compute, and operational effort at your target scale, including the cost of re-indexing when the embedding model changes.
Measure end-to-end quality too. For a RAG or agent system, a small drop in recall can matter less than adding a reranking step, or a great deal more if the missing documents are the ones that answer the question.
The future of vector databases
- Hybrid search becomes the default: The combination of vector similarity, keyword matching, and metadata filtering in a single query interface is emerging as the standard.
- Consolidation into existing databases: PostgreSQL with pgvector, MongoDB Atlas, and Elasticsearch now offer vector search, and for many workloads the question is shifting from which vector database to adopt to whether a separate one is needed at all.
- GPU-accelerated retrieval: GPU indexing, demonstrated at billion-vector scale in Meta's FAISS research (Johnson et al., 2017), is increasingly used to cut index build times and raise query throughput for very large collections.
- Multimodal storage: Vector databases are evolving to store and search across text, images, audio, and video.
- Agent memory: As autonomous AI agents proliferate, vector databases are becoming the memory layer for agent state.
The choice of vector database is no longer a tactical technology decision. It is an architectural decision that shapes what an organisation can build with AI for years to come.
Frequently asked questions
Sources and further reading
- Lewis, P. et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Facebook AI Research, UCL, NYU, 2020. arxiv.org/abs/2005.11401
- Morris, J. X. et al. Text Embeddings Reveal (Almost) As Much As Text. Cornell, EMNLP 2023. arxiv.org/abs/2310.06816
- MLCommons. ANN Benchmarks for Vector Search. mlcommons.org
- European Commission. The EU AI Act. 2025. digital-strategy.ec.europa.eu
- Malkov, Y. and Yashunin, D. Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs. 2016. arxiv.org/abs/1603.09320
- Jégou, H., Douze, M., and Schmid, C. Product Quantization for Nearest Neighbor Search. IEEE TPAMI, 2011. doi.org/10.1109/TPAMI.2010.57
- Subramanya, S. J. et al. DiskANN: Fast Accurate Billion-point Nearest Neighbor Search on a Single Node. Microsoft Research, NeurIPS 2019. papers.nips.cc
- Johnson, J., Douze, M., and Jégou, H. Billion-scale Similarity Search with GPUs. Meta AI, 2017. arxiv.org/abs/1702.08734
- Aumüller, M., Bernhardsson, E., and Faithfull, A. ANN-Benchmarks: A Benchmarking Tool for Approximate Nearest Neighbor Algorithms. 2018. arxiv.org/abs/1807.05614
- Park, J. S. et al. Generative Agents: Interactive Simulacra of Human Behavior. Stanford and Google, 2023. arxiv.org/abs/2304.03442
- Packer, C. et al. MemGPT: Towards LLMs as Operating Systems. UC Berkeley, 2023. arxiv.org/abs/2310.08560
- Greshake, K. et al. Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. 2023. arxiv.org/abs/2302.12173
- Zou, W. et al. PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models. 2024. arxiv.org/abs/2402.07867
