Tiny Local RAG: LangChain + LangGraph + Ollama (Markdown Files)

Let’s build a tiny Retrieval-Augmented Generation (RAG) system.

You point it at a folder of Markdown files (for example, your blog posts). It builds a vector index. You ask questions. It answers using your own content — with citations.

It runs locally. No cloud required.

And yes — it works on macOS, Linux, and Windows.

What Is RAG (in simple words)?

RAG means:

Retrieve relevant text first → then generate an answer.

Instead of hoping the model “remembers” your files (it doesn’t), we:

Split your files into small chunks
Convert each chunk into numbers (embeddings)
Store those numbers in a vector database
When you ask a question:
- embed the question
- retrieve the most relevant chunks
- give them to the model as context
The model answers using those chunks

That’s it.

Minimal Architecture

Markdown files
   ↓
Split into chunks
   ↓
Embeddings (meaning as numbers)
   ↓
Vector store (Chroma on disk)
   ↓
LangGraph workflow:
   retrieve → answer
   ↓
Answer with citations

LangChain handles:

Embeddings
Vector store
Prompt templating

LangGraph handles:

Clean workflow structure

Step 1 — Install Ollama (All Platforms)

You must have Ollama installed and running.

Then pull models:

ollama pull nomic-embed-text
ollama pull qwen2.5:7b

You can use a different chat model if you prefer.

Step 2 — Create a Project

macOS / Linux

mkdir tiny-rag
cd tiny-rag
python3 -m venv .venv
source .venv/bin/activate

Windows (PowerShell)

mkdir tiny-rag
cd tiny-rag
py -m venv .venv
.\.venv\Scripts\Activate.ps1

If activation fails:

Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser

Step 3 — Install Python Packages

All platforms:

pip install -U pip
pip install langchain langchain-core langchain-community langchain-chroma langchain-ollama langgraph chromadb

Project Structure

tiny-rag/
  rag_app.py
  files/
    example1.md
    example2.md
  chroma_db/   (created automatically)

Put some Markdown files in files/.

The Working Code

Create rag_app.py:

from __future__ import annotations

import os
from pathlib import Path
from typing import List, TypedDict

from langchain_community.document_loaders import TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_chroma import Chroma
from langchain_ollama import OllamaEmbeddings, ChatOllama
from langchain_core.documents import Document
from langchain_core.messages import HumanMessage, SystemMessage
from langgraph.graph import StateGraph, END


class RAGState(TypedDict, total=False):
    question: str
    retrieved: List[Document]
    answer: str
    show_sources: bool


# -------- File loading --------

def load_markdown_files(folder: str) -> List[Document]:
    docs: List[Document] = []
    base = Path(folder)

    for path in base.rglob("*.md"):
        loader = TextLoader(str(path), encoding="utf-8")
        loaded = loader.load()
        for d in loaded:
            d.metadata["source"] = str(path)
        docs.extend(loaded)

    if not docs:
        raise ValueError(f"No .md files found in: {folder}")

    return docs


def split_docs(docs: List[Document]) -> List[Document]:
    splitter = RecursiveCharacterTextSplitter(
        chunk_size=900,
        chunk_overlap=120,
    )
    return splitter.split_documents(docs)


def get_vectorstore(persist_dir: str) -> Chroma:
    embed_model = os.getenv("OLLAMA_EMBED_MODEL", "nomic-embed-text")
    embeddings = OllamaEmbeddings(model=embed_model)
    return Chroma(
        collection_name="md_rag",
        embedding_function=embeddings,
        persist_directory=persist_dir,
    )


def build_or_update_index(files_dir: str, persist_dir: str):
    docs = load_markdown_files(files_dir)
    chunks = split_docs(docs)

    vs = get_vectorstore(persist_dir)
    vs.delete_collection()
    vs = get_vectorstore(persist_dir)

    vs.add_documents(chunks)
    # Chroma 0.4+ persists automatically once persist_directory is set —
    # no explicit vs.persist() call needed (it was removed from the API).


def retrieve(vs: Chroma, question: str, k: int = 4):
    return vs.similarity_search(question, k=k)


# -------- LangGraph nodes --------

def node_retrieve(state: RAGState) -> RAGState:
    vs = get_vectorstore(os.getenv("CHROMA_DIR", "chroma_db"))
    top_k = int(os.getenv("TOP_K", "4"))
    docs = retrieve(vs, state["question"], k=top_k)
    state["retrieved"] = docs

    if state.get("show_sources"):
        print("\n--- RETRIEVED SOURCES ---\n")
        for i, d in enumerate(docs, start=1):
            src = d.metadata.get("source", "unknown")
            snippet = d.page_content.strip().splitlines()[0][:140]
            print(f"[{i}] {src}")
            print(f"    {snippet}...\n")
        print("-------------------------\n")

    return state


def format_context(docs: List[Document]) -> str:
    lines = []
    for i, d in enumerate(docs, start=1):
        src = d.metadata.get("source", "unknown")
        lines.append(f"[{i}] SOURCE: {src}\n{d.page_content.strip()}\n")
    return "\n".join(lines)


def node_answer(state: RAGState) -> RAGState:
    context = format_context(state["retrieved"])

    llm = ChatOllama(
        model=os.getenv("OLLAMA_CHAT_MODEL", "qwen2.5:7b"),
        temperature=0.2,
    )

    system = SystemMessage(content=(
        "Answer using ONLY the provided context. "
        "If the answer is not in the context, say you don't know. "
        "Cite snippets using [1], [2], etc."
    ))

    user = HumanMessage(content=(
        f"QUESTION:\n{state['question']}\n\n"
        f"CONTEXT:\n{context}"
    ))

    resp = llm.invoke([system, user])
    state["answer"] = resp.content.strip()
    return state


def build_graph():
    g = StateGraph(RAGState)
    g.add_node("retrieve", node_retrieve)
    g.add_node("answer", node_answer)
    g.set_entry_point("retrieve")
    g.add_edge("retrieve", "answer")
    g.add_edge("answer", END)
    return g.compile()


# -------- CLI --------

def main():
    import argparse

    parser = argparse.ArgumentParser()
    parser.add_argument("--index", action="store_true")
    parser.add_argument("--files", default="files")
    parser.add_argument("--question", default="")
    parser.add_argument("--show-sources", action="store_true")
    args = parser.parse_args()

    persist_dir = os.getenv("CHROMA_DIR", "chroma_db")

    if args.index:
        print("Building index...")
        build_or_update_index(args.files, persist_dir)
        print("Index complete.")
        return

    if not args.question:
        raise SystemExit("Provide --question or run with --index first.")

    graph = build_graph()
    result = graph.invoke({
        "question": args.question,
        "show_sources": args.show_sources
    })

    print("\n--- ANSWER ---\n")
    print(result["answer"])
    print("\n--------------\n")


if __name__ == "__main__":
    main()

Run It

Build index:

python rag_app.py --index --files files

Ask a question:

python rag_app.py --question "What topics are covered?" --show-sources

You will see:

Retrieved snippets
Then the grounded answer

That’s RAG in action.

Troubleshooting

❗ “No .md files found”

Make sure your files/ folder contains Markdown files.

❗ Ollama model not found

Run:

ollama pull nomic-embed-text
ollama pull qwen2.5:7b

❗ Slow answers

Use a smaller chat model and reduce retrieval size:

export OLLAMA_CHAT_MODEL="qwen2.5:7b"
export TOP_K=3

Windows:

$env:OLLAMA_CHAT_MODEL="qwen2.5:7b"
$env:TOP_K="3"

What You Built

You now have:

A local RAG system
A persistent vector index
A LangGraph workflow
Source visibility
Cross-platform compatibility

And it fits on one screen of code.

That’s the kind of minimal system I like.

RAG is not about Markdown. It’s about loaders.

One small exercise before you go, because it proves the point.

Little Exercise: Add `.txt` or `.pdf` Support

Right now, our system only loads:

for path in base.rglob("*.md"):

That’s intentional — we kept it minimal.

But RAG works with any text-based content.

Let’s extend it.

Supporting `.txt` Files (Very Easy)

Change the loader section to:

for path in base.rglob("*"):
    if path.suffix.lower() in [".md", ".txt"]:
        loader = TextLoader(str(path), encoding="utf-8")
        loaded = loader.load()
        for d in loaded:
            d.metadata["source"] = str(path)
        docs.extend(loaded)

That’s it.

Now your RAG system supports:

Markdown
Plain text notes
Logs
Exported documentation

No other change required.

Supporting `.pdf` Files (Slightly More Interesting)

First install a PDF loader:

pip install pypdf

Then update the imports:

from langchain_community.document_loaders import TextLoader, PyPDFLoader

Modify the file-loading loop:

for path in base.rglob("*"):
    if path.suffix.lower() == ".md" or path.suffix.lower() == ".txt":
        loader = TextLoader(str(path), encoding="utf-8")

    elif path.suffix.lower() == ".pdf":
        loader = PyPDFLoader(str(path))

    else:
        continue

    loaded = loader.load()
    for d in loaded:
        d.metadata["source"] = str(path)
    docs.extend(loaded)

Rebuild your index:

python rag_app.py --index

Now your RAG can search PDFs too.

What Changed?

Only the loader.

Not:

The vector store
The embeddings
The retrieval
The graph
The answering logic

That’s the beauty of this architecture.

Why This Is a Good Exercise

It teaches three key lessons:

RAG is modular.
Loaders define what you can search.
The orchestration layer doesn’t care what format your documents are.

Once you understand that, you can index:

Your Obsidian vault
Research papers
Code repositories
Legal documents
Meeting transcripts

Same structure. Different loader.

Retrieval Quality Controls

Tune chunk size and overlap per document type.
Store metadata (source, section, timestamp) with each chunk.
Return top-k citations alongside the answer.
Reject answers when retrieval confidence is low.
Re-index incrementally when source files change.

Tiny Local RAG: LangChain + LangGraph + Ollama (Markdown Files)

📚 This post is part of the "AI Orchestration" series

Tiny Local RAG: LangChain + LangGraph + Ollama (Markdown Files)

What Is RAG (in simple words)?

Minimal Architecture

Step 1 — Install Ollama (All Platforms)

Step 2 — Create a Project

macOS / Linux

Windows (PowerShell)

Step 3 — Install Python Packages

Project Structure

The Working Code

Run It

Troubleshooting

❗ “No .md files found”

❗ Ollama model not found

❗ Slow answers

What You Built

Little Exercise: Add `.txt` or `.pdf` Support

Supporting `.txt` Files (Very Easy)

Supporting `.pdf` Files (Slightly More Interesting)

What Changed?

Why This Is a Good Exercise

Retrieval Quality Controls

References

Citation

Tiny Local RAG: LangChain + LangGraph + Ollama (Markdown Files)

📚 This post is part of the "AI Orchestration" series

Tiny Local RAG: LangChain + LangGraph + Ollama (Markdown Files)

What Is RAG (in simple words)?

Minimal Architecture

Step 1 — Install Ollama (All Platforms)

Step 2 — Create a Project

macOS / Linux

Windows (PowerShell)

Step 3 — Install Python Packages

Project Structure

The Working Code

Run It

Troubleshooting

❗ “No .md files found”

❗ Ollama model not found

❗ Slow answers

What You Built

Little Exercise: Add .txt or .pdf Support

Supporting .txt Files (Very Easy)

Supporting .pdf Files (Slightly More Interesting)

What Changed?

Why This Is a Good Exercise

Retrieval Quality Controls

Stay Ahead in AI, Machine Learning & Python

You're in

References

Citation

Little Exercise: Add `.txt` or `.pdf` Support

Supporting `.txt` Files (Very Easy)

Supporting `.pdf` Files (Slightly More Interesting)