Retrieval-augmented generation with vector databases

Generated by Claude through GigAI 0.1.11.10 on 2026-10-11.

Concept: Retrieval-augmented generation · also covers Vector databases · Step 16 of 34 in AI forward deployed engineer · version 1

Practice this

On this page

Retrieval-augmented generation supplies a model with relevant passages from an external corpus at request time, so answers are grounded in data the model was never trained on and can cite it. The system has two halves: an offline pipeline that parses, chunks, embeds and indexes documents, and an online path that retrieves, ranks and assembles context for generation. Most quality problems in RAG are retrieval problems, so the engineering effort goes into chunking, search and measurement.

Read more

How it is done in practice

A team begins with the customer's real documents and a list of real questions, then labels which passages answer each one; that labeled set drives every later decision. Ingestion runs as a repeatable pipeline keyed on content hashes so changed documents are re-chunked and re-embedded while unchanged ones are skipped, and deletions propagate to the index. The embedding model and its version are stored with every vector, since changing the model means re-embedding the corpus. Online, the service applies tenant and access filters, runs hybrid search, reranks, and assembles a cited context block. Recall at k, latency per stage and the rate of declined answers are tracked on dashboards, and index parameters are tuned against the labeled set instead of by intuition.

Technologies

pgvectorQdrantPineconeWeaviateElasticsearchOpenSearchBM25Cohere RerankVoyage AI embeddingsOpenAI embeddingsLlamaIndexLangChain

Diagram

flowchart LR
  subgraph Offline ingestion
    A[Source documents] --> B[Parse and clean]
    B --> C[Chunk and add context note]
    C --> D[Embed]
    D --> E[(Vector index)]
    C --> F[(Keyword index)]
  end
  subgraph Online query
    Q[User question] --> G[Embed query and apply access filters]
    G --> E
    Q --> F
    E --> H[Fuse candidates]
    F --> H
    H --> I[Rerank and keep top k]
    I --> J[Assemble cited context]
    J --> K[Model generates grounded answer]
  end
flowchart LR
  subgraph Offline ingestion
    A[Source documents] --> B[Parse and clean]
    B --> C[Chunk and add context note]
    C --> D[Embed]
    D --> E[(Vector index)]
    C --> F[(Keyword index)]
  end
  subgraph Online query
    Q[User question] --> G[Embed query and apply access filters]
    G --> E
    Q --> F
    E --> H[Fuse candidates]
    F --> H
    H --> I[Rerank and keep top k]
    I --> J[Assemble cited context]
    J --> K[Model generates grounded answer]
  end

Code

sql
CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE chunks (
  id bigserial PRIMARY KEY,
  tenant_id int NOT NULL,
  doc_id text NOT NULL,
  body text NOT NULL,
  embedding vector(1024) NOT NULL
);

-- HNSW index for cosine distance
CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);

-- raise recall at query time at the cost of speed
SET hnsw.ef_search = 100;

SELECT id, doc_id, body, embedding <=> $1 AS distance
FROM chunks
WHERE tenant_id = $2
ORDER BY embedding <=> $1
LIMIT 20;

Subtopics

RAG architecture: parametric and non-parametric memory

The framing behind the pattern

The original RAG work described a model with two memories: parametric memory in the weights and non-parametric memory in a dense index of passages, queried by a retriever and consumed by a generator. That framing still guides design choices. Knowledge that changes, is private, or must be attributable belongs in the index, where it can be updated or deleted without retraining. When a corpus is small enough to fit in the window, placing it there directly, with prompt caching, can be simpler and more accurate than retrieval, so check corpus size before building a pipeline.

Chunking and contextual retrieval

Restoring context that chunking removes

Splitting a document strips each chunk of its surroundings: a passage stating that revenue grew three percent no longer says which company or quarter. Contextual retrieval addresses this by having a model write a short chunk-specific note that situates the passage within the whole document, then prepending that note to the chunk before it is embedded and before it is added to the keyword index. Anthropic reports a substantial drop in retrieval failures from this step, with further gains when combined with reranking. Prompt caching of the full document keeps the per-chunk generation cost manageable.

Hybrid search and reranking

Adding a second ranking stage

Fused dense and lexical results are a candidate set, not a final ordering. A reranker, usually a cross-encoder that scores the query and each passage together, is more accurate than embedding similarity because it reads both texts jointly, but it is too slow to run over a whole corpus. The standard arrangement is two stages: retrieve a wide set, on the order of a hundred or more candidates, rerank them, and pass only the top few to the generator. This raises precision in the window and reduces tokens spent on marginal passages, at the price of added latency per query.

Vector indexes and vector databases

Approximate nearest neighbor indexes

Exact search compares the query vector with every stored vector, which is accurate but linear in corpus size. Approximate indexes trade a little recall for speed. HNSW builds a layered graph and gives a strong speed and recall balance, with slower builds and more memory; its main knobs are graph connectivity, build-time candidate list size and a query-time search width that raises recall when increased. IVFFlat clusters vectors into lists and probes a subset; it builds faster and uses less memory but should be created after data is loaded. The distance metric, cosine, inner product or Euclidean, must match how the embedding model was trained.

Postgres extension or dedicated engine

pgvector adds a vector column type, distance operators and both index types to Postgres, keeping embeddings next to relational data with transactions, joins and existing backups. Dedicated engines such as Qdrant organize data into collections of points, each holding one or more vectors plus a JSON payload, and offer payload indexes for filtered search, quantization to shrink memory, sharding and replication. Filtering is the usual deciding factor: tenant or permission filters applied after an approximate search can return too few results, so the engine must combine filter and vector search during traversal.

Grounded generation and retrieval evaluation

Measuring the two halves separately

Evaluate retrieval and generation independently so a failure can be assigned to one of them. For retrieval, build a set of questions with known relevant passages and track recall at k, the share of questions whose needed passage appears in the top k, along with rank-sensitive measures such as mean reciprocal rank. For generation, check faithfulness, meaning every claim is supported by a retrieved passage, and answer relevance. Instruct the model to quote or cite passage ids and to decline when the context does not contain the answer, then verify cited ids exist and support the claim. Enforce document permissions at retrieval time, never by asking the model to withhold text.

Sources

Related lessons

Practice this
Structured outputs and tool callingAgent architectures and agentic workflows with agent SDKs