Vector databases have moved from research curiosity to production infrastructure component in the space of two years, driven almost entirely by the adoption of retrieval-augmented generation (RAG) architectures for LLM-powered applications. The premise is straightforward: embed documents as dense vectors, store them in a database that supports efficient nearest-neighbour search, and at query time embed the user's query and retrieve the most semantically similar documents as context for the language model.
The implementation is less straightforward. Production RAG systems face challenges that toy examples do not: embedding freshness as the knowledge base grows, retrieval quality degradation as vector counts increase, latency requirements for interactive applications, and operational overhead that compounds as the number of collections and namespaces grows. This article covers the design and operations of vector databases in production AI systems.
When You Actually Need a Vector Database
Vector databases are the right tool for semantic search at scale: finding documents that are conceptually similar to a query, even when they do not share keywords. The core use cases are RAG (retrieving relevant context for LLM prompts), semantic search in knowledge bases, recommendation systems where similarity is in embedding space rather than rule space, and deduplication of large text corpora where identical-meaning but differently-worded documents need to be identified.
A relational database with pgvector is sufficient for many vector search use cases at moderate scale. PostgreSQL with the pgvector extension supports exact and approximate nearest-neighbour search on vector columns, integrates naturally with relational data, and eliminates the operational complexity of maintaining a separate vector database. For production RAG systems with up to a few million documents, pgvector on a well-provisioned Postgres instance is often the right choice.
Dedicated vector databases become necessary when: the vector collection exceeds tens of millions of documents, latency requirements demand sub-10ms retrieval, or the search use case requires features beyond basic vector similarity (hybrid search combining vector and keyword retrieval, metadata filtering at query time, multi-tenancy with namespace isolation). Understand which of these requirements your system actually has before adopting dedicated vector infrastructure.
Vector Database Comparison
Weaviate is the most feature-complete open-source vector database for enterprise RAG applications: native multi-tenancy for SaaS use cases, hybrid search combining vector and BM25 keyword search, GraphQL and REST APIs, and a well-maintained Python client. It runs well on Kubernetes and has a managed cloud offering. The trade-off is operational complexity — Weaviate's schema management and multi-tenancy configuration require careful design.
Qdrant is an excellent choice for high-performance use cases where raw retrieval speed and memory efficiency are priorities. Written in Rust, it has lower memory overhead than JVM-based alternatives and consistent sub-millisecond retrieval latency at moderate scale. Qdrant's filtering is particularly good — filtered vector search (retrieve vectors similar to the query AND matching a metadata predicate) is well-optimised. The managed cloud offering is newer than Weaviate's.
Pinecone is the simplest managed offering and the fastest to production: serverless pricing, no cluster management, and a simple API. The trade-off is cost at scale — Pinecone pricing becomes expensive for very large collections, and the serverless model has latency variance that dedicated infrastructure avoids. For teams that want to get to production quickly and can accept the cost structure, Pinecone removes most of the operational friction.
Vector Database Selection Guide
- Small to medium RAG (<5M vectors): PostgreSQL + pgvector
- Fast production, managed: Pinecone serverless
- Enterprise, multi-tenant, hybrid search: Weaviate
- High-performance, self-hosted: Qdrant
- Existing Elasticsearch stack: Elasticsearch dense vector field
Indexing and Retrieval Quality
Embedding model selection has a larger impact on retrieval quality than vector database choice. The embedding model determines how well semantic similarity in the vector space corresponds to semantic similarity in the document domain. General-purpose embedding models (text-embedding-3-large from OpenAI, e5-large, nomic-embed-text) perform well on general knowledge retrieval. Domain-specific tasks — medical text, legal documents, technical specifications — often benefit from domain-fine-tuned embedding models.
Chunking strategy — how you split source documents into the units that get embedded — is the most frequently underestimated retrieval quality factor. Documents split into chunks that are too small lose context needed for the embedding to be semantically meaningful. Chunks that are too large include irrelevant context that dilutes the semantic signal. For most text documents, 200 to 500 token chunks with 20% overlap (adjacent chunks share some content) is a practical starting point. Test different chunking strategies against your specific document types using retrieval evaluation metrics.
Hybrid search — combining vector similarity search with keyword search (BM25) and merging the results — consistently outperforms pure vector search for knowledge base RAG applications. Pure vector search finds semantically similar documents but can miss documents that contain specific technical terms or proper nouns that are important but do not significantly affect the embedding. Hybrid search addresses this by ensuring that exact keyword matches are not penalised by embedding distance. Weaviate and Elasticsearch have native hybrid search; other databases can achieve it by combining results from a separate keyword index.
Production Performance
Query latency for vector search consists of two components: embedding the query (calling the embedding model) and the vector search itself. For OpenAI text-embedding-3-small, embedding latency is typically 50–200ms via the API. For self-hosted embedding models (e5-small, minilm), embedding latency is 5–20ms on CPU, 1–5ms on GPU. If query latency is critical, the decision between hosted and self-hosted embedding models is as important as the vector database selection.
Approximate nearest neighbour (ANN) indexing — HNSW is the most common algorithm — trades recall for speed. An HNSW index configured for 95% recall will find the 95% most relevant documents, not necessarily the 100% most relevant, but will do it 10–100× faster than exhaustive search. For RAG applications, 95–99% recall is typically sufficient — missing 1% of relevant documents has negligible impact on response quality. Configure index parameters for your recall-latency trade-off requirement, not for maximum recall.
Memory usage scales with vector dimensions × number of vectors × bytes per dimension. A collection of 1 million 1536-dimensional vectors (OpenAI text-embedding-3-small output size) at float32 requires approximately 6 GB of RAM just for the vectors, before index overhead. Quantisation (storing vectors at lower precision — 8-bit instead of 32-bit) reduces memory by 4× with modest impact on retrieval quality. Enable quantisation for large collections where memory cost is a constraint.
Operations and Maintenance
Embedding freshness — keeping the vector index current as the source knowledge base changes — is the primary ongoing operational task for RAG systems. Documents added, updated, or deleted in the source system must be reflected in the vector index. Design a synchronisation pipeline that processes document change events and updates the vector index as changes occur, rather than relying on periodic full re-indexing that creates freshness windows.
Index migrations — changing the embedding model, adjusting chunk sizes, or restructuring collections — require re-embedding the entire knowledge base, which can take hours for large collections and incurs significant API cost if using hosted embedding models. Maintain the current and new index simultaneously during migrations, validate retrieval quality on the new index before switching traffic, and retire the old index only after the new one is confirmed correct.
Multi-tenancy in RAG systems requires isolation between tenants at the vector database level. A SaaS application where each customer's documents should only be retrievable for that customer's queries needs either separate collections per tenant (clean isolation, higher operational overhead) or namespace/filter-based isolation within shared collections (lower overhead, requires correct filter application on every query without exception). The choice depends on scale: separate collections are manageable for tens of tenants, impractical for thousands.
