RAG Explained: How to Give an AI a Library Card

RAG stands for Retrieval-Augmented Generation. It is a method for helping an artificial intelligence system answer questions using information retrieved from documents, databases, websites, notes, or other external sources.

The simplest explanation is this:

A standard language model answers from memory. A RAG system searches its reference material first, then answers with the relevant pages open on the desk.

RAG turns an AI from a closed-book improviser into something closer to an open-book researcher.

Why RAG Exists

Large language models are powerful, but they have important limitations.

Their training knowledge eventually becomes outdated. They do not automatically know what is inside your private documents. They can confuse similar facts, invent plausible-sounding information, and struggle when important material is buried inside a large archive.

Imagine asking an AI:

What does the Deadwater skate shop plan say about live music?

A general-purpose model has never seen your private business plan. Without RAG, you would need to paste the relevant material into the conversation yourself.

With RAG, the system searches your stored documents, finds the passages about music events, places those passages into the model’s prompt, and asks the model to answer using that evidence.

The AI is not magically remembering the document. It is being handed the right material at the moment it is needed.

The Basic RAG Pipeline

A RAG system usually has two major phases.

The first phase prepares the knowledge:

Documents
   ↓
Extract text
   ↓
Split the text into chunks
   ↓
Create embeddings
   ↓
Store the chunks and embeddings

The second phase answers a question:

User question
   ↓
Convert the question into an embedding
   ↓
Retrieve relevant chunks
   ↓
Optionally rerank them
   ↓
Place the best chunks into the prompt
   ↓
Generate an answer

The defining feature is that retrieval happens before generation.

Step 1: Collect the Documents

The source material for a RAG system can include almost anything:

• PDFs
• webpages
• Google Docs
• emails
• research papers
• product manuals
• meeting notes
• business plans
• transcripts
• support tickets
• legal documents
• database records

A personal creative archive might include essays, product descriptions, brand guidelines, project notes, research, design briefs, and technical documentation.

Each collection becomes part of a searchable AI library.

Step 2: Extract and Clean the Content

Before a system can search a document, it must extract usable content from it.

A PDF may contain headings, paragraphs, tables, captions, footnotes, diagrams, page numbers, and scanned images. The system needs to identify which elements are meaningful and preserve their structure.

This stage is called document parsing.

Parsing quality matters enormously. If a clean financial table is transformed into alphabet soup, the retrieval system will faithfully retrieve alphabet soup.

Scanned documents may also require optical character recognition, or OCR, before their text can be indexed.

Step 3: Divide the Content Into Chunks

Large documents are usually broken into smaller pieces called chunks.

Suppose a business plan contains 30,000 words. Sending the entire document to the model every time someone asks a question would be slow, expensive, and distracting.

Instead, the document might be divided into chunks of a few hundred words or tokens.

For example:

Chunk 1:
Deadwater will sell locally manufactured skateboard decks...

Chunk 2:
The retail space will use the visual language of an old hardware store...

Chunk 3:
The shop may host small hardcore shows, art exhibitions,
premieres, and community events...

When the user asks about music events, the system retrieves the third chunk.

Good chunking tries to preserve meaningful units such as complete paragraphs, sections, policy clauses, table rows, or question-and-answer pairs.

Bad chunking can slice an idea in half:

Chunk 1 ends:
The shop may host small hardcore

Chunk 2 begins:
shows, art exhibitions, and community events.

That is the textual equivalent of cutting a sandwich directly through the plate.

Many systems use overlapping chunks so that a small portion of the previous section is repeated in the next one. This reduces the chance that an important idea will disappear across a boundary.

Step 4: Create Embeddings

An embedding is a numerical representation of meaning.

A sentence is converted into a long list of numbers:

"Deadwater will host hardcore shows."

→ [0.037, -0.214, 0.661, 0.092, ...]

The numbers are not designed to be read by humans. Their purpose is to place related ideas near one another in a mathematical space.

For example, these phrases should produce embeddings that are relatively close:

live punk events
hardcore concerts
music performances at the shop

They use different words, but they express similar ideas.

This allows the system to search by meaning rather than relying only on exact keyword matches.

Step 5: Store the Embeddings

The embeddings and their associated text chunks are stored in a searchable index, often called a vector database.

A stored record might look conceptually like this:

{
  "text": "Deadwater may host hardcore shows and art exhibitions.",
  "embedding": [0.037, -0.214, 0.661],
  "metadata": {
    "document": "Deadwater Business Plan",
    "section": "Community Events",
    "page": 14,
    "project": "Deadwater"
  }
}

The metadata is almost as important as the embedding.

It can be used to restrict a search:

Only search Deadwater documents
Only search policies published after 2025
Only search English-language files
Only search documents this user is allowed to access

Without metadata, the AI may rummage through every filing cabinet at once.

Step 6: Retrieve Relevant Information

When a user asks a question, the system converts that question into an embedding and compares it with the stored document embeddings.

Suppose the user asks:

What kinds of events will Deadwater hold?

The system may return results like these:

Community Events section          similarity 0.91
Art Exhibition Plan               similarity 0.86
Store Layout and Stage Area       similarity 0.78
Deck Manufacturing                similarity 0.42

The system then selects the strongest results and sends them to the language model.

This is often called top-k retrieval, where k represents the number of chunks selected.

Too few chunks may omit important evidence. Too many may bury the answer beneath a compost heap of vaguely related text.

Semantic Search and Keyword Search

Vector search is excellent for matching ideas, but it is not perfect for every kind of information.

Consider a product code:

DW-847-BLK

A semantic embedding may not handle that identifier particularly well. Exact keyword search is often better for names, dates, model numbers, legal clause numbers, abbreviations, and unusual technical terms.

Traditional keyword search is strong at finding exact language.

Vector search is strong at finding paraphrases and related concepts.

A good production system often combines the two in a method called hybrid search:

Keyword search
        +
Vector search
        ↓
Combined ranking

This gives the system both a librarian who understands the topic and a clerk who remembers the exact serial number.

Reranking the Results

The first retrieval step is designed to be fast, but it may return passages that are only loosely relevant.

A reranking model can examine the candidates more carefully.

Retrieve 30 possible chunks
            ↓
Reranker evaluates each result
            ↓
Keep the best 5
            ↓
Send those 5 to the language model

The initial search casts a wide net. The reranker inspects what wriggled into it.

Reranking is especially useful when the archive is large, the documents contain similar terminology, or the question requires precise evidence.

Building the Prompt

Once the system has selected the best passages, it places them into the language model’s prompt.

A simplified version might look like this:

You are an assistant answering questions about Deadwater.

Use only the supplied sources.
If the sources do not contain the answer, say so.
Cite the document and section used.

QUESTION:
What types of events will Deadwater host?

SOURCE 1:
Document: Deadwater Business Plan
Section: Community Events
Text: Deadwater may host small hardcore shows,
art exhibitions, skateboard premieres, and workshops.

SOURCE 2:
Document: Deadwater Store Concept
Section: Flexible Retail Space
Text: Fixtures will be movable so the shop can become
an event space after retail hours.

The model might answer:

Deadwater plans to host small hardcore shows, art exhibitions, skateboard premieres, and workshops. Its movable fixtures would allow the retail floor to become an event space after closing.

The model generates the prose, but the factual content comes from retrieved evidence.

RAG Does Not Eliminate Hallucinations

RAG can reduce hallucinations, but it cannot eliminate them.

A RAG system can fail because:

• the correct document was never indexed
• text extraction failed
• chunking separated important context
• retrieval selected the wrong passages
• the model ignored the supplied material
• the documents contradicted one another
• the source was outdated
• the source itself was incorrect
• the question required calculation rather than retrieval

A trustworthy system must be able to say:

The available sources do not answer that question.

That sentence is often more useful than a polished hallucination wearing a tiny academic hat.

RAG Versus Fine-Tuning

RAG and fine-tuning are frequently confused, but they solve different problems.

RAG gives the model reference information at the time a question is asked.

It is useful for:

• private documents
• changing facts
• product catalogs
• policies
• research archives
• technical documentation
• organizational knowledge

Fine-tuning changes how the model behaves by training it on examples.

It is useful for:

• tone
• formatting
• classification
• consistent response patterns
• specialized task behavior

A useful distinction is:

RAG teaches the model what to look at. Fine-tuning teaches it how to behave.

A system may use both. RAG can provide project facts and source material, while fine-tuning can encourage a consistent style or output structure.

RAG Versus Long-Context Prompting

Modern language models can accept increasingly large prompts. That raises an obvious question: why not simply paste every document into the prompt?

For small archives, that may be the simplest solution.

For larger archives, it has disadvantages:

• higher cost
• slower responses
• more irrelevant information
• context limits
• repeated transmission of the same documents
• reduced attention to details buried in the middle

RAG acts as a gatekeeper. It sends the model the relevant shelves rather than loading the entire Library of Congress into a wheelbarrow.

RAG Versus Database Queries

This distinction is especially important for financial, scientific, and business systems.

Suppose you ask:

How much did I spend on groceries last month?

That is primarily a structured database question. The system should filter transactions by date and category, then calculate the total.

Now consider:

According to my adviser’s notes, why did we decide to reduce restaurant spending?

That is a RAG question because the answer lives inside unstructured prose.

A capable assistant may combine both approaches:

Structured query
   → calculate restaurant spending

RAG retrieval
   → find the adviser’s explanation

Language model
   → combine the numbers and the supporting context

RAG is excellent for retrieving prose and evidence. Databases are better for precise calculations, filtering, and aggregation.

A Simple RAG Example

Suppose we begin with three document chunks:

documents = [
    "Deadwater will sell skateboard decks and selected hardware.",
    "Deadwater may host hardcore shows, art exhibitions, and workshops.",
    "The store interior will resemble an old neighborhood hardware store."
]

During indexing, each chunk is converted into an embedding and stored:

for document in documents:
    vector = embedding_model.embed(document)

    vector_database.store(
        vector=vector,
        text=document
    )

When a user asks a question, that question is also embedded:

question = "Will the shop have music events?"

question_vector = embedding_model.embed(question)

results = vector_database.search(
    vector=question_vector,
    top_k=3
)

The strongest result may be:

Deadwater may host hardcore shows, art exhibitions, and workshops.

The system then builds a prompt:

context = "\n".join(result.text for result in results)

prompt = f"""
Answer the question using the supplied context.
If the answer is not present, say that you do not know.

Context:
{context}

Question:
{question}
"""

The language model may generate:

Yes. The plans say Deadwater may host hardcore shows, along with art exhibitions and workshops.

That is the basic RAG machine.

More Advanced RAG Systems

A production-quality system often includes additional stages:

Documents
   ↓
Parsing
   ↓
Cleaning
   ↓
Chunking
   ↓
Embedding
   ↓
Indexing
   ↓
Query rewriting
   ↓
Metadata filtering
   ↓
Hybrid retrieval
   ↓
Reranking
   ↓
Context assembly
   ↓
Answer generation
   ↓
Citation verification

Each stage exists because someone discovered a fascinating new way for the simpler version to fail.

Query Rewriting

Users often ask vague follow-up questions such as:

What about the music part?

The system may rewrite that internally as:

What does the Deadwater business plan say about hosting music events?

The rewritten query is easier to search against.

Permissions

A workplace archive may contain public manuals, legal documents, customer information, employee records, and executive notes.

The retrieval system must enforce permissions before any content is shown to the model.

Do not retrieve everything and ask the model to keep the secrets secret. Security belongs in the retrieval layer.

Citations

A strong RAG system tracks each passage back to its source, including the document title, page number, section heading, date, and original location.

This allows users to verify claims and distinguish source material from the model’s interpretation.

Multimodal RAG

RAG is not limited to text. A multimodal system can retrieve images, diagrams, audio transcripts, video segments, charts, scanned pages, and product photographs.

A creative archive could connect an image with its design brief, product listing, related blog post, print specifications, and source files.

Graph RAG

Standard RAG retrieves passages based mainly on similarity.

Graph RAG also models relationships between people, projects, places, concepts, and events.

For example:

Pope of Love
   ├── created → Holy Ghost Cosmology
   ├── appears in → Experimental Church
   ├── teaches → The Nine Gates
   └── associated with → Carbonated Thoughts

This is useful when a question requires following connections across several documents rather than retrieving a single passage.

Agentic RAG

Traditional RAG performs one search.

Agentic RAG allows the AI to perform a sequence of research actions:

Interpret the question
   ↓
Search one collection
   ↓
Inspect the results
   ↓
Search another collection
   ↓
Query a database
   ↓
Compare the evidence
   ↓
Generate an answer

This is more powerful, but every additional decision point gives the machine another opportunity to wander into the shrubbery.

How RAG Systems Are Evaluated

A RAG system should be evaluated in two separate areas.

The first is retrieval quality.

Did the system find the passages that contained the answer? Were they ranked near the top? Did irrelevant passages crowd them out?

The second is answer quality.

Given the retrieved evidence, did the model answer correctly? Did it cite the proper sources? Did it omit important information? Did it acknowledge uncertainty?

A model cannot answer from evidence that retrieval failed to deliver. This is why RAG debugging should separate retrieval failures from generation failures.

A Practical Technology Stack

A simple self-hosted RAG project might use:

Language:         Python
API:              FastAPI
Document parser:  PyMuPDF or Docling
Vector database:  PostgreSQL with pgvector or Qdrant
Keyword search:   PostgreSQL full-text search or OpenSearch
Interface:        A simple web chat

Frameworks such as LlamaIndex, LangChain, Haystack, and Semantic Kernel can accelerate development.

They are useful, but it is still important to understand the underlying pipeline. Otherwise, debugging becomes a séance conducted through six layers of abstraction.

The Smallest Useful RAG Project

A good first project is an “Ask My Project Archive” assistant.

You could upload:

• business plans
• essays
• brand guides
• product descriptions
• design notes
• technical documentation
• research materials

Then you could ask:

What decisions have we made about the visual identity?

Which projects mention the Monad?

What features have already been proposed?

Which ideas appear across multiple projects?

The first version only needs file upload, text extraction, chunking, embeddings, vector search, prompt assembly, citations, and a chat interface.

That small project teaches nearly every important RAG concept while producing something genuinely useful.

The Central Idea

RAG is not simply “chat with PDFs.”

It is an architecture for connecting a language model to evidence.

A strong RAG system answers four questions:

1. What information is available?
2. Which pieces are relevant to this question?
3. Which sources should be trusted?
4. How should the model answer without going beyond the evidence?

The language model is the narrator. Retrieval is the research department. Metadata is the filing system. Reranking is the editor. Citations are the receipts.

Remove any one of them, and the machine becomes noticeably more theatrical than reliable.