Elena' s AI Blog

Tiny Local RAG: LangChain + LangGraph + Ollama (Markdown Files)

22 Sep 2026 (updated: 05 Oct 2026) / 17 minutes to read

Elena Daehnhardt

Generated by ChatGPT-5. Prompt: Programmer in a bright modern office with multiple screens.


TL;DR:
  • A ~150-line local RAG pipeline over a folder of Markdown files, with no cloud calls. LangChain's RecursiveCharacterTextSplitter chunks the files at 900 characters with 120 characters of overlap; Ollama's nomic-embed-text model produces the embeddings; Chroma persists the index to disk; a two-node LangGraph workflow (retrieve then answer) queries it with qwen2.5:7b and cites each retrieved chunk as [1], [2]. Swapping the loader — TextLoader for .md and .txt, PyPDFLoader for .pdf — is the only change needed to index a different document type; the vector store, embeddings, retrieval, and graph are untouched.

Previous: Part 18 — DeepSeek R1 With Ollama

Next: Part 30 — Operational Polish: Human Reports and Draft Preview Endpoints

Where this fits:RAG Explore the AI Landscape →

Tiny Local RAG: LangChain + LangGraph + Ollama (Markdown Files)

Let’s build a tiny Retrieval-Augmented Generation (RAG) system.

You point it at a folder of Markdown files (for example, your blog posts). It builds a vector index. You ask questions. It answers using your own content — with citations.

It runs locally. No cloud required.

And yes — it works on macOS, Linux, and Windows.


What Is Retrieval-Augmented Generation (RAG)?

Retrieval-Augmented Generation (RAG) is an architecture that retrieves relevant text from your own documents at query time and passes it to a language model as context, so the model answers from evidence you supplied rather than from its training data. Lewis et al. introduced the term in 2020, and the shape has barely changed since.

Retrieve relevant text first → then generate an answer.

Instead of hoping the model “remembers” your files (it doesn’t), we:

  1. Split your files into small chunks
  2. Convert each chunk into numbers (embeddings)
  3. Store those numbers in a vector database
  4. When you ask a question:

    • embed the question
    • retrieve the most relevant chunks
    • give them to the model as context
  5. The model answers using those chunks

That’s it.


Minimal Local RAG Architecture: Chunks, Embeddings, Chroma and LangGraph

Markdown files
   ↓
Split into chunks
   ↓
Embeddings (meaning as numbers)
   ↓
Vector store (Chroma on disk)
   ↓
LangGraph workflow:
   retrieve → answer
   ↓
Answer with citations

LangChain handles:

  • Embeddings
  • Vector store
  • Prompt templating

LangGraph handles:

  • Clean workflow structure

The step-by-step setup, the full code, running and troubleshooting, and extending the loader follow below.

Still in this guide

  • Step 1 — Install Ollama and Pull the Embedding and Chat Models
  • Step 2 — Create the Project and Virtual Environment
  • Step 3 — Install the LangChain, LangGraph and Chroma Packages
  • Project Structure
  • Full rag_app.py: Loader, Splitter, Chroma Store and LangGraph Workflow
  • Running the Indexer and Asking a Question
  • Troubleshooting Common Local RAG Errors
  • What the Tiny RAG Pipeline Gives You
  • Extending the Loader: Adding `.txt` and `.pdf` Support
  • Why Only the Loader Changed: RAG Component Boundaries
  • Three Lessons: RAG Modularity, Loaders and Format Independence
  • Retrieval Quality Controls for a Production Local RAG

You've hit a Deep Dive tutorial.

I spend dozens of hours researching, coding, and breaking things to write these guides. This content is free, but reserved for my subscriber community. Drop your email below to unlock this guide (and all past/future deep dives):

Already a subscriber? Use the magic link from your last newsletter, or reset your password.

New subscribers get an inbox mail: Set a password to unlock articles. The form does not log you in — use the same email afterwards.

desktop bg dark

About Elena

Elena, a PhD in Computer Science, simplifies AI concepts and helps you use machine learning.

Citation
Elena Daehnhardt. (2026) 'Tiny Local RAG: LangChain + LangGraph + Ollama (Markdown Files)', daehnhardt.com, 22 September 2026. Available at: https://daehnhardt.com/blog/2026/09/22/tiny-local-rag/
All Posts