Why I wanted a local AI lab
It has been a stranger few months than usual for anyone who builds on top of rented AI models, and by September the pattern was hard to ignore.
On 12 June, a US export-control directive forced Anthropic to pull Claude Fable 5 and Mythos 5 offline worldwide. The directive itself only targeted foreign nationals, but Anthropic disabled both models for every customer, everywhere, to stay compliant — a frontier model that plenty of developers had already wired into their workflows simply evaporated within days of launch. (The directive was lifted later that month, but the outage still made the point: access to a rented model can vanish on a government’s schedule, not yours.)
On 1 June, GitHub Copilot moved to usage-based billing. The flat monthly fee gained a meter: Copilot plans now bill against GitHub AI Credits, with extra usage charged once your monthly allowance runs out.
And while all that was happening, OpenCode — an open-source, model-agnostic coding agent that is perfectly happy talking to a local Ollama server — reached 7% developer adoption in JetBrains’ 2026 developer survey, with 42% of developers aware of it despite no big company name behind it.
The common thread is hard to miss. The cloud model you build on can be revoked, repriced, or rate-limited on someone else’s timetable; an open-weight model sitting on your own disk cannot. That has always been the quiet argument for a local AI lab, and this summer made it loud.
The good news is that the local AI ecosystem has hit a practical tipping point at exactly the right moment. Models that would have required a data centre two years ago now run on a mid-range laptop, and the tooling to serve, interact with, and augment those models has become genuinely approachable.
This post walks through a complete local AI lab: a model engine, a web interface that feels like ChatGPT, and a private RAG pipeline that lets your AI answer questions grounded in your own documents.
I run this stack on an Apple M1 MacBook, so where the steps differ from other platforms I have called that out explicitly.
A companion post covers the security picture — specifically, what changes when you add autonomous agents to this setup.
What we’re actually building
| Component |
Job |
| Ollama |
Loads and runs the LLM |
| Open WebUI |
Gives you the chat interface |
| Embedding model |
Turns document chunks and queries into vectors |
| RAG |
Finds relevant chunks before asking the LLM |
| Docker |
Runs Open WebUI in an isolated container |
Here is how those pieces fit together — worth pinning down before you install anything, because it is easy to assume Docker is somehow running your model. It is not. Open WebUI is the interface and orchestration layer; Ollama does the actual inference.
Your Mac
┌──────────────────────────────────────────────────┐
│ │
│ Browser │
│ │ │
│ ▼ │
│ Open WebUI (Docker) │
│ │ │
│ ├──────────────► Ollama │
│ │ │ │
│ │ ▼ │
│ │ Local LLM │
│ │ │
│ └──► Knowledge base │
│ │ │
│ ▼ │
│ Embedding model │
│ │ │
│ ▼ │
│ Vector retrieval │
│ │ │
│ └────────► context ─────► Local LLM │
│ │
└──────────────────────────────────────────────────┘
The whole stack runs offline once everything is downloaded. You can pull the Wi-Fi cable and it keeps working — more on what that actually means, and does not mean, later in this post.
By the end of this post, you will have:
- Llama 3.2 running locally through Ollama
- a ChatGPT-style Open WebUI interface, reachable at
http://localhost:3000
- persistent chats and settings that survive a container restart
- a private knowledge base built from your own documents
- a test proving retrieval actually works, not just that it looks installed
- a test proving the whole stack still works with no internet connection
Phase 1: The model engine — Ollama
🔒 Subscribe to keep reading.
Phase 2: The web interface — Open WebUI
🔒 Subscribe to keep reading.
Phase 3: Private RAG — your AI on your documents
🔒 Subscribe to keep reading.
Phase 4: Prove the lab
🔒 Subscribe to keep reading.
Phase 5: Operate the lab
🔒 Subscribe to keep reading.
How much hardware do you actually need?
🔒 Subscribe to keep reading.
What “private” actually means
🔒 Subscribe to keep reading.
Where agents change the threat model
🔒 Subscribe to keep reading.
Quick-start checklist for a local AI lab
🔒 Subscribe to keep reading.
References
🔒 Subscribe to keep reading.
You've hit a Deep Dive tutorial.
I spend dozens of hours researching, coding, and breaking things to write these guides. This content is free, but reserved for my subscriber community. Drop your email below to unlock this guide (and all past/future deep dives):
Full content temporarily unavailable — refresh in a moment
Already a subscriber? Use the magic link from your last newsletter, or reset your password.
Log in to unlock
New subscribers get an inbox mail: Set a password to unlock articles. The form does not log you in — use the same email afterwards.
