Vector Databases: Indexes, Filtering, and Retrieval Operations

Vector databases store embedding vectors and serve approximate nearest-neighbor (ANN) search—usually with metadata filters, multi-tenant isolation, and operational controls for upserts, deletes, and reindexing. Managed systems from vendors such as Pinecone and Weaviate compete on filters and ops. They are retrieval infrastructure, not embedding trainers and not full RAG applications. This guide owns ANN index operations, filtering and tenancy patterns, and reliability concerns for vector stores. How vectors are learned and evaluated stays in embeddings. Faithfulness, ACL-aware corpus design, and answer grounding stay in RAG. Serving silicon and cluster economics deepen in AI infrastructure.

If you treat a vector database as a magical “AI memory,” you will skip the contracts that make search correct: embedding model version, distance metric, filter semantics, and recall measurement. The database accelerates neighbor lookup; it does not invent relevance.

What a vector database guarantees

At minimum, a vector database accepts vectors of fixed dimension, indexes them, and returns top-k neighbors for a query vector under a declared metric (cosine, dot, L2). Most production systems also store payload metadata, support filtered search, and expose upsert/delete APIs. Guarantees differ on consistency after writes, exact vs approximate recall, and whether filters are applied pre-, post-, or during graph traversal.

Read the product’s consistency model. “Eventually searchable” after upsert is common; designing UX that assumes instant global visibility will create support tickets. Durability (is the vector on disk?) differs from searchability (is it in the ANN structure?). Ops runbooks must name both.

A vector database is not a system of record for documents. Keep canonical text and permissions in primary stores; keep vectors as derived artifacts keyed by stable document/chunk IDs. When embeddings models change, you rebuild vectors—you should not lose source truth.

Compared with relational search, vector DBs optimize geometric neighborhoods. They are weak at arbitrary joins and strong at “find similar.” Hybrid architectures pair them with keyword indexes and application-side fusion—patterns owned end-to-end by RAG, executed partly here.

ANN index families and recall/latency trade-offs

Exact kNN (brute force) is the ground truth for small collections and evaluation. ANN indexes trade a controlled recall loss for latency and memory wins at scale. HNSW-style graph indexes offer strong recall/latency with RAM-heavy footprints. IVF-style inverted file indexes partition space and probe a subset of lists—tunable via nprobe. Disk-oriented indexes spill vectors or graph layers to SSD for larger-than-memory corpora at higher tail latency.

Parameters are product decisions: efSearch, M, nprobe, quantization codecs. Raising recall usually raises latency and cost. Publish an operating curve for your collection: recall@k vs p95 latency vs memory. Do not copy blog defaults for a corpus two orders of magnitude different.

Quantized indexes (PQ, SQ, binary) shrink memory and can destroy neighborhoods if misapplied. Validate recall on stratified samples after every codec change. Quantization is an index version, not a free toggle.

Build time matters. Large HNSW builds are CPU- and memory-intensive; plan maintenance windows or blue/green index builds. Online inserts into graphs can degrade quality over time—some systems recommend periodic rebuilds. Measure insert-path recall drift.

Metadata filtering and hybrid queries

Filters (tenant, locale, product line, ACL group) are where vector databases meet real applications. Pre-filtering restricts the candidate set before ANN; post-filtering retrieves neighbors then drops non-matches—can return fewer than k results; inline filtered traversal tries to keep recall under predicates. Wrong choice yields empty results or leaked neighbors.

Filter cardinality changes performance. High-selectivity filters (one tenant among thousands) can defeat graph indexes if the engine is not filter-aware. Test filtered recall, not only unfiltered. Synthetic queries with realistic predicate distributions beat vanity demos.

Hybrid queries combine dense ANN with sparse/keyword scores. Some engines integrate sparse vectors or BM25-like signals; others require application-side fusion (RRF, weighted sums). Fusion policy belongs with RAG quality ownership; the database must expose the primitives—dense hits, sparse hits, payloads—with stable ranking hooks.

Range filters on timestamps and numeric fields enable freshness and business rules. Push deterministic constraints to filters; do not hope the embedding encodes “updated this week.”

Ingestion, upserts, and tombstones

Ingestion pipelines encode documents via the embedding service, then upsert vectors with payloads and IDs. Idempotent upserts keyed by chunk ID prevent duplicates when jobs retry. Partial batch failures need replay without double-billing neighbors.

Deletes should remove searchability promptly. Tombstones that linger create ghost results. Compaction and garbage collection reclaim space; schedule them and watch query latency during GC.

Schema evolution—new payload fields, dimension changes—requires explicit migrations. Dimension changes imply new collections; never mix dimensions in one index. Payload-only changes are easier but still need backfills for old rows.

Backpressure matters when re-embedding entire corpora. Rate-limit upserts to protect query SLOs. Prefer writing a shadow collection, validating recall, then atomically switching aliases.

Sharding, replication, and consistency

Shard by tenant, by hash of ID, or by collection. Tenant sharding simplifies isolation and noisy-neighbor control; hash sharding balances load for global corpora. Cross-shard queries increase fan-out latency—design query planners accordingly.

Replication provides availability and read scale. Understand whether replicas are searchable during catch-up. Split-brain after network partitions is rare but catastrophic if two writers accept conflicting upserts—prefer single-writer primaries or conflict-free ID schemes.

Multi-region active-active vector search is hard: embedding model versions and index parameters must match. Many teams keep regional indexes fed from a shared embedding bus rather than synchronous cross-region ANN.

Consistency SLAs should be stated in product language: “searchable within N seconds for 99% of upserts.” Measure it. Silent lag produces “I uploaded the doc and chat can’t find it” incidents blamed on the LLM.

Multi-tenant isolation patterns

Collection-per-tenant maximizes isolation and simplifies deletes; it can explode operational overhead at thousands of tenants. Shared collections with mandatory tenant filters are cheaper and riskier—every query path must enforce the predicate. Defense in depth: filter + separate API keys + audit logs.

Noisy neighbors: one tenant’s bulk reindex can starve others’ queries. Quotas on upsert QPS, disk, and concurrent builds are mandatory in shared clusters. Align with infrastructure multi-tenant GPU lessons—same isolation mindset, different resource.

Encryption and key management for payloads and vectors follow data classification. Vectors of sensitive text are sensitive. At-rest encryption and private networking are baseline for enterprise offerings.

Observability of recall regressions

Latency histograms, error rates, and CPU/RAM are necessary but insufficient. Track recall proxies: golden query suites with expected IDs, filtered empty-result rates, and distribution shifts in top-score histograms. A sudden score collapse often means metric mismatch, unnormalized vectors, or wrong model version—not “ANN broke.”

Correlate upsert lag, GC duration, and query p99. Capacity dashboards should show memory headroom before HNSW builds fail mid-way. Alert on collection size vs configured RAM assumptions.

Version everything visible to query: index build ID, embedding model ID, metric, and filter schema hash. On-call should answer “what changed?” from telemetry, not folklore.

Load tests must include filtered queries and realistic payload sizes. Unfiltered microbenchmarks lie about production.

Cost drivers: memory vs disk indexes

RAM dominates HNSW economics: dimension × count × bytes × graph overhead × replicas. Disk indexes trade dollars for latency. Quantization reduces RAM and can raise compute per query. Reserved capacity vs burst upsert windows change unit economics.

Re-embedding costs often exceed query costs during migrations—include embedding API/GPU time in TCO. Alias cutovers beat in-place mutations for large estates.

Right-size top-k and over-fetch for filters. Fetching 200 neighbors to keep 10 after post-filters wastes capacity. Prefer filter-aware indexes when predicates are selective.

Where the DB ends and the RAG app begins

The database returns neighbors and payloads. The RAG application owns query rewriting, hybrid fusion policy, rerankers, citation assembly, faithfulness checks, and user-facing ACL beyond crude tenant IDs. Crossing that line—stuffing prompt logic into DB stored procedures—creates untestable systems.

Agents may call vector search as a tool; permissioning those calls is an AI agents concern. The DB still enforces tenant filters as a hard floor.

LLMs do not replace indexes. Dumping corpora into a large language model context is not a vector database strategy; it is a cost and privacy strategy failure for large corpora.

Operational runbooks

Blue/green reindex: build collection B with new model or parameters; run golden suite; switch alias; keep A for rollback TTL. Capacity incident: shed bulk upserts, raise cache, reduce efSearch temporarily with known recall impact, page owners. Suspected leak: freeze queries, rotate credentials, audit filterless admin paths.

Document who can create collections, who can drop them, and who can change aliases. Alias flips are production releases.

Anti-patterns

One shared index with optional tenant filters. Mixing embedding models in a collection. Tuning ANN to fix bad chunking. Post-filter-only on highly selective ACL. No golden suite. Reindexing in place on Friday without rollback. Treating vendor “99% recall” claims as your filtered recall. Storing canonical documents only inside the vector DB.

Worked sketches

SaaS knowledge base: collection-per-tenant or shared+mandatory tenant filter; HNSW in RAM until cost forces disk tier for cold tenants; alias migration on embedding bump.

Global product catalog: hash-sharded IVF or disk ANN; strong metadata filters for locale/currency; hybrid lexical for SKUs; payload pointers to PIM—not full HTML in vectors.

Internal code search: high upsert churn; prioritize incremental indexing and fast deletes; evaluate filtered recall by repo ACL.

Selecting engines and deployment modes

Managed vector services optimize speed-to-market; self-hosted engines optimize control and data residency. Evaluate filtered query support, backup/restore, observability hooks, and dimension limits—not only raw QPS screenshots. Proof-of-concept with your embedding dimensionality and predicate mix.

Embedded libraries inside app processes suit single-tenant prototypes; they rarely meet multi-tenant ops needs. Separate the query service early if you expect tenancy.

Exit drills: export IDs, payloads, and vectors; rebuild elsewhere within a defined RTO. If you cannot export, you do not own your retrieval layer.

Capacity planning checklist

Estimate active vectors, dimension, bytes per vector with overhead, replica count, peak QPS, filtered query fraction, upsert bursts, and reindex frequency. Add headroom for graph build spikes. Align budgets with embedding encode capacity so reindexes are not blocked on CPU while the DB sits idle.

Review quarterly: collections unused, tenants oversized, indexes never rebuilt after million-scale inserts. Compaction debt is a reliability risk.

Security notes for vector stores

Admin APIs that bypass tenant filters are privileged; MFA and break-glass only. Audit every filterless query. Sanitize payloads displayed back into prompts to avoid injection via retrieved text—application responsibility with DB as the retrieval source.

Backups contain vectors and metadata; protect them like database dumps of sensitive content. Redact locally when sharing snapshots with vendors.

Closing

Vector databases turn embedding geometry into operable search: indexes, filters, tenancy, and measurable recall under change. Keep embedding training in the embeddings guide; keep grounded generation in RAG; keep silicon economics in infrastructure. Version collections like releases. Measure filtered recall. Isolate tenants. Plan reindexes as migrations, not afterthoughts.

Index lifecycle and aliases

Treat collections as immutable artifacts when parameters change. Build, validate, alias-switch, retain predecessor for rollback. Aliases decouple client configuration from physical collection names so applications pin a logical endpoint while ops rotate underlying indexes.

Lifecycle states help coordination: building, validating, live, draining, archived. Only one live alias per product surface unless you intentionally shadow traffic. Draining stops writes, finishes reads, then archives. Archived collections still cost money—set TTLs.

Automated pipelines should refuse to alias-switch when golden suite deltas exceed thresholds. Human override exists for emergencies and must be logged. Silent switches recreate the worst habits of undocumented model updates in LLM stacks.

Document rebuild triggers: embedding model bump, dimension change, metric change, catastrophic recall drop, or scheduled defragmentation after heavy deletes. Not every payload field addition needs a rebuild.

Client libraries and API contracts

Clients must send normalized vectors when the index expects them. Mismatched normalization is a leading cause of “random” neighbors. Encode query and document sides with identical preprocessing libraries—share a package between indexer and online path.

Batch query APIs reduce overhead under fan-out; streaming upserts need idempotency keys. Timeouts should be shorter than user-facing budgets so the RAG layer can fall back to lexical search.

Pagination over ANN results is poorly defined in many engines—prefer larger top-k with application truncation. Cursor semantics that assume stable global order will surprise you under concurrent upserts.

Health checks should verify not only process uptime but also that a canary vector returns an expected neighbor. Process-up and recall-down is a common failure mode after bad deploys.

Disaster recovery

Backups must capture vectors, payloads, index parameters, and alias maps. Restoring vectors without the embedding model ID documentation leaves you unable to query correctly after a rebuild. Store model IDs beside snapshots.

RPO/RTO targets differ for search indexes versus primary document stores. It is often acceptable to rebuild from source documents if embedding capacity exists—sometimes faster than restoring a corrupt graph. Practice both paths.

Region failure playbooks: fail over reads to a replica region with known lag; pause writes or buffer upserts; communicate searchable delay to product surfaces. Do not invent cross-region synchronous ANN unless you have tested it under partition.

Team ownership matrix

Platform owns cluster capacity, upgrades, and multi-tenant quotas. Search/ML owns embedding versions and golden suites. Security owns tenant isolation reviews. Product owns UX for empty filtered results. Ambiguity produces orphan collections and surprise bills.

Change tickets for alias flips should list suite links, memory delta, and rollback collection. Treat them with the same seriousness as database migrations that change primary keys.

Failure gallery

Query vectors float32, index int8 without documented projection—neighbors look plausible and wrong. Tenant filter omitted on an internal debug tool—cross-tenant leakage. Post-filter ACL with top-k=10 on a rare predicate—empty answers blamed on the LLM. Reindex without pausing writes—torn reads across old and new embeddings. Vendor upgrade changes default metric—silent quality cliff.

Each incident traces to a missing invariant: metric parity, mandatory filters, filtered recall tests, write fencing during migration, or pinned defaults. Write invariants into the collection charter before scaling QPS.

When p99 spikes, check GC, concurrent HNSW builds, and payload bloat before buying larger nodes. Many “capacity” problems are operational collisions, not raw vector count.

Ship vector database changes as versioned migrations with suites and rollbacks. That is how ANN infrastructure earns a place beside relational datastores in the enterprise stack—not by marketing “AI memory,” but by operable, measurable neighbor search under filters and tenancy.

Keep the mental model small: embeddings create coordinates; vector databases search neighborhoods under constraints; RAG decides what those neighbors mean for a user answer. When those layers stay separated, each can evolve without breaking the others.

References and further reading

• Malkov, Y. A., & Yashunin, D. A. (2018). Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs.
• Jégou, H., Douze, M., & Schmid, C. (2011). Product quantization for nearest neighbor search. IEEE TPAMI.
• Johnson, J., Douze, M., & Jégou, H. (2017). Billion-scale similarity search with GPUs.