Tiny Local RAG: LangChain + LangGraph + Ollama (Markdown Files)

Let’s build a tiny Retrieval-Augmented Generation (RAG) system.

You point it at a folder of Markdown files (for example, your blog posts). It builds a vector index. You ask questions. It answers using your own content — with citations.

It runs locally. No cloud required.

And yes — it works on macOS, Linux, and Windows.

What Is Retrieval-Augmented Generation (RAG)?

Retrieval-Augmented Generation (RAG) is an architecture that retrieves relevant text from your own documents at query time and passes it to a language model as context, so the model answers from evidence you supplied rather than from its training data. Lewis et al. introduced the term in 2020, and the shape has barely changed since.

Retrieve relevant text first → then generate an answer.

Instead of hoping the model “remembers” your files (it doesn’t), we:

1. Split your files into small chunks
2. Convert each chunk into numbers (embeddings)
3. Store those numbers in a vector database

4. When you ask a question:

   • embed the question
   • retrieve the most relevant chunks
   • give them to the model as context
5. The model answers using those chunks

That’s it.

Minimal Local RAG Architecture: Chunks, Embeddings, Chroma and LangGraph

Markdown files
   ↓
Split into chunks
   ↓
Embeddings (meaning as numbers)
   ↓
Vector store (Chroma on disk)
   ↓
LangGraph workflow:
   retrieve → answer
   ↓
Answer with citations

LangChain handles:

• Embeddings
• Vector store
• Prompt templating

LangGraph handles:

• Clean workflow structure

The step-by-step setup, the full code, running and troubleshooting, and extending the loader follow below.

Step 1 — Install Ollama and Pull the Embedding and Chat Models

🔒 Subscribe to keep reading.

Step 2 — Create the Project and Virtual Environment

🔒 Subscribe to keep reading.

Step 3 — Install the LangChain, LangGraph and Chroma Packages

🔒 Subscribe to keep reading.

Project Structure

🔒 Subscribe to keep reading.

Full rag_app.py: Loader, Splitter, Chroma Store and LangGraph Workflow

🔒 Subscribe to keep reading.

Running the Indexer and Asking a Question

🔒 Subscribe to keep reading.

Troubleshooting Common Local RAG Errors

🔒 Subscribe to keep reading.

What the Tiny RAG Pipeline Gives You

🔒 Subscribe to keep reading.

Extending the Loader: Adding .txt and .pdf Support

🔒 Subscribe to keep reading.

Why Only the Loader Changed: RAG Component Boundaries

🔒 Subscribe to keep reading.

Three Lessons: RAG Modularity, Loaders and Format Independence

🔒 Subscribe to keep reading.

Retrieval Quality Controls for a Production Local RAG

🔒 Subscribe to keep reading.