Build Local RAG Over Your Own Files

Retrieval-augmented generation is a search problem wearing a language model as a hat. You cut your documents into pieces, turn each piece into a vector, find the pieces closest to the question, and paste them into the prompt. The model does the last step only. Everything that determines whether the answer is any good happens before the model sees anything.

Two local components do the work: Ollama's /api/embed endpoint for the vectors, and Chroma for storing and searching them. Both run on your machine, neither needs a key, and the whole pipeline below is about sixty lines of Python.

Three moving parts, and only one of them is the model

Ollama's embeddings page names three models it recommends for this job: embeddinggemma, qwen3-embedding and all-minilm. Pull the first one:

ollama pull embeddinggemma

The endpoint is POST http://localhost:11434/api/embed, described in the docs as creating "vector embeddings representing the input text". The smallest working call:

curl http://localhost:11434/api/embed -d '{ "model": "embeddinggemma", "input": "Why is the sky blue?" }'

The response carries model, an embeddings array of arrays, and three timing and accounting fields: total_duration, load_duration and prompt_eval_count. Passing an array to input instead of a string batches the call and returns one vector per item, which is how you index a folder without paying the model-load cost per chunk.

Two request parameters matter more than they look. dimensions sets the output vector length. truncate defaults to true and silently cuts inputs that exceed the model's context window. Hold that second one, because it is about to become the most important sentence in this part.

Two rules from the same page are worth following without argument: "Use the same embedding model for both indexing and querying" and "Use cosine similarity for most semantic search use cases."

EmbeddingGemma is 622MB with a 2K token window

Ollama's library card lists embeddinggemma at 300M parameters, 622MB on disk, and a 2K context window, with a note that "This model requires Ollama v0.11.10 or later".

Google's own page for the same model says 308M parameters, not 300M. It is a rounding difference rather than a contradiction, but it tells you the Ollama card is a summary rather than the spec sheet, and the Ollama card omits the fields you actually need.

Google publishes those. Output dimensions are flexible: "Customize your output dimensions from 768 to 128" using Matryoshka Representation Learning. The context is "2K token". The model can "Run it on less than 200MB of RAM with quantization", and it was trained across "over 100 languages".

The 2K window is the number that governs your design. It is not a suggestion, and because Ollama's truncate parameter defaults to true, a chunk longer than 2K tokens gets embedded from its first 2K tokens and the rest is discarded without an error. Your vector store fills up, every query returns results, and half of each document was never indexed.

Set "truncate": false during a first indexing run if you would rather get an error than a quiet half-index. Then fix the chunker.

Chunking against a 2K window, not a 128K one

Chat models advertise 40K, 128K and 256K windows. Embedding models do not. The one you just pulled stops at 2K, and every chunk you produce has to fit inside that number after tokenization, not after character counting.

Practical shape for the chunker:

• Split on structure first. Headings, then paragraphs. A chunk that spans two unrelated sections retrieves badly for both.
• Keep chunks well under the 2K ceiling so the tokenizer's estimate never becomes the deciding factor.
• Overlap adjacent chunks by a paragraph so a fact split across a boundary appears whole in at least one of them.
• Store the source path and position in metadata. You will want to show the reader where an answer came from, and you cannot reconstruct it later.

There is no documented optimal chunk size, and anyone who gives you one without a citation is guessing. What is documented is the ceiling, so design against that.

Chroma stores vectors you computed yourself

Install it:

pip install chromadb

The default client is in-memory, and Chroma's docs say what that means: "any data you ingest will be lost when your program terminates." For anything you intend to keep, use the persistent client:

import chromadb
client = chromadb.PersistentClient(path="/path/to/save/to")

The docs describe path as "where Chroma will store its database files on disk, and load them on start". Omit it and the default is .chroma, which is a fine way to lose an index by running your script from a different directory.

Chroma will embed documents for you if you let it. Do not let it, because that would use a different model from the one you queried with and break the same-model rule above. Pass your own vectors instead:

collection.add(
  ids=["id1", "id2", "id3"],
  embeddings=[[1.1, 2.3, 3.2], [4.5, 6.9, 4.4], [1.1, 2.3, 3.2]],
  documents=["doc1", "doc2", "doc3"],
  metadatas=[{"chapter": 3, "verse": 16}, {"chapter": 3, "verse": 5}, {"chapter": 29, "verse": 11}],
)

Chroma's documentation is explicit about the behaviour: when you supply your own embeddings alongside documents, "Chroma will store both as-is without re-embedding the documents."

Querying takes the same treatment. Embed the question through /api/embed, then search by vector rather than by text:

results = collection.query(
  query_embeddings=[[11.1, 12.1, 13.1]],
  n_results=5,
  include=["documents", "metadatas", "distances"]
)

The result is a QueryResult with ids, embeddings, documents, uris, metadatas, distances and included. Results come back in what Chroma calls "column-major form", grouped per input query as nested arrays, so a single question means you want results["documents"][0] rather than results["documents"].

Two filters are worth building in from the start. where={"page": 10} filters on metadata, and where_document={"$contains": "search string"} filters on the text itself. Filtering to one source file before searching is usually cheaper and more accurate than asking the vector search to figure it out.

Ollama recommends cosine, Chroma defaults to l2

These two pages give different instructions and neither mentions the other.

Ollama's embeddings page says to "Use cosine similarity for most semantic search use cases". Chroma's collection configuration page lists three values for the HNSW space parameter, with l2 as the default: it "measures absolute geometric distance between vectors", against cosine which "measures only the angle between vectors (ignoring magnitude)" and ip.

Follow both docs literally and you get L2 while one of them told you to use cosine.

There is a detail that defuses this specific case, and it comes from Ollama's page too: "The /api/embed endpoint returns L2-normalized (unit-length) vectors." For vectors that all sit on the unit sphere, ordering by L2 distance and ordering by cosine similarity produce the same ranking. So the default is not silently wrecking your results as long as your vectors come from that endpoint unmodified.

It stops being harmless the moment you mix in vectors from somewhere that does not normalize, or truncate dimensions and renormalize inconsistently. Set it explicitly and stop thinking about it:

collection = client.create_collection(
  name="my-collection",
  configuration={"hnsw": {"space": "cosine"}}
)

Retrieval fails quietly, so read the chunks first

The failure mode that wastes the most time is a confident wrong answer that looks like a model problem and is a retrieval problem.

Before you tune a prompt or swap a chat model, print what came back. If the five chunks in results["documents"][0] do not contain the answer, no prompt engineering will produce it, and a bigger model will simply state the wrong thing more fluently.

A diagnostic order that works:

1. Print the retrieved text. Wrong chunks means a chunking, indexing or metric problem, not a model problem.
2. Check distances. If the top hit and the fifth hit have near-identical distances, the query did not discriminate and your chunks are probably too long or too uniform.
3. Re-embed one document with "truncate": false. An error here proves you have been indexing prefixes of documents rather than documents.
4. Confirm the same model on both sides. Query vectors from a different model than the index is a silent, total failure, and Ollama's docs flag it as the first rule for a reason.
5. Check the chat model's context. Five chunks plus a system prompt plus a question can exceed a 4096-token default without warning, and the first thing dropped is usually your retrieved context.

That last step connects back to the rest of this series. A RAG pipeline pushes far more tokens into the prompt than chat does, so the context window you set when you installed Ollama is a load-bearing part of this design rather than a nice-to-have.

What this buys you and what it does not

The buys are real. Nothing leaves the machine, there is no per-query cost, the index is a directory you can back up, and it works with no network.

The costs are also real. You are writing and maintaining a chunker, an indexer and a retrieval loop that a hosted product would give you for free. NotebookLM does the same job with an upload button. Choose local when the documents cannot leave, when the corpus is large enough that per-query pricing matters, or when you want the retrieval logic under your own control. Those are good reasons. Curiosity about whether it works is a fine reason too, as long as you know which one you are acting on.

If the answer at the end of the pipeline is weak rather than wrong, the model is the last thing to change and part 3 covers how to swap one in.