RAG in plain English (and a minimal implementation)

The idea

An LLM doesn't know your company handbook. RAG fixes that without retraining: when a question comes in, you search your documents for the most relevant passages and paste them into the prompt, telling the model to answer only from them.

Retrieval finds the right pages; generation writes the answer. Most quality problems are retrieval problems.

The pipeline

1. 1Ingest: extract text from PDFs, docs and web pages.
2. 2Chunk: split into passages of a few hundred words, on headings or paragraphs, with a little overlap.
3. 3Embed: turn each chunk into a vector with an embedding model and store it with its source.
4. 4Retrieve: embed the question and fetch the top 5–10 nearest chunks.
5. 5Generate: send the chunks plus the question to the LLM with a grounded prompt.

The table and the query

With Postgres and the pgvector extension, retrieval is one SQL query. Set the vector size to match your embedding model.

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE chunks (
  id        bigserial PRIMARY KEY,
  source    text NOT NULL,      -- e.g. "handbook.pdf#p12"
  content   text NOT NULL,
  embedding vector(1024)        -- match your embedding model
);

-- nearest 6 chunks to the question embedding ($1)
SELECT source, content
FROM chunks
ORDER BY embedding <=> $1
LIMIT 6;

The grounded prompt

Answer the question using only the sources below.
Cite sources in square brackets, like [handbook.pdf#p12].
If the sources don't contain the answer, say you don't know.

<sources>
[handbook.pdf#p12] ...chunk text...
[policy.docx#3] ...chunk text...
</sources>

Question: {{question}}