Tiny Local RAG: LangChain + LangGraph + Ollama (Markdown Files)
Let’s build a tiny Retrieval-Augmented Generation (RAG) system.
You point it at a folder of Markdown files (for example, your blog posts).
It builds a vector index.
You ask questions.
It answers using your own content — with citations.
It runs locally.
No cloud required.
And yes — it works on macOS, Linux, and Windows.
What Is Retrieval-Augmented Generation (RAG)?
Retrieval-Augmented Generation (RAG) is an architecture that retrieves relevant text from your own documents at query time and passes it to a language model as context, so the model answers from evidence you supplied rather than from its training data. Lewis et al. introduced the term in 2020, and the shape has barely changed since.
Retrieve relevant text first → then generate an answer.
Instead of hoping the model “remembers” your files (it doesn’t), we:
- Split your files into small chunks
- Convert each chunk into numbers (embeddings)
- Store those numbers in a vector database
-
When you ask a question:
- embed the question
- retrieve the most relevant chunks
- give them to the model as context
- The model answers using those chunks
That’s it.
Minimal Local RAG Architecture: Chunks, Embeddings, Chroma and LangGraph
Markdown files
↓
Split into chunks
↓
Embeddings (meaning as numbers)
↓
Vector store (Chroma on disk)
↓
LangGraph workflow:
retrieve → answer
↓
Answer with citations
LangChain handles:
- Embeddings
- Vector store
- Prompt templating
LangGraph handles:
The step-by-step setup, the full code, running and troubleshooting, and extending the loader follow below.
Still in this guide
- Step 1 — Install Ollama and Pull the Embedding and Chat Models
- Step 2 — Create the Project and Virtual Environment
- Step 3 — Install the LangChain, LangGraph and Chroma Packages
- Project Structure
- Full rag_app.py: Loader, Splitter, Chroma Store and LangGraph Workflow
- Running the Indexer and Asking a Question
- Troubleshooting Common Local RAG Errors
- What the Tiny RAG Pipeline Gives You
- Extending the Loader: Adding `.txt` and `.pdf` Support
- Why Only the Loader Changed: RAG Component Boundaries
- Three Lessons: RAG Modularity, Loaders and Format Independence
- Retrieval Quality Controls for a Production Local RAG
You've hit a Deep Dive tutorial.
I spend dozens of hours researching, coding, and breaking things to write these guides. This content is free, but reserved for my subscriber community. Drop your email below to unlock this guide (and all past/future deep dives):
Full content temporarily unavailable — refresh in a moment
Already a subscriber? Use the magic link from your last newsletter, or reset your password.
Log in to unlock
New subscribers get an inbox mail: Set a password to unlock articles. The form does not log you in — use the same email afterwards.