Architecting the Great Divide: How to Run a Multi-LLM Stack as a Team of One

If you looked at the mid-July AI releases, one thing became blindingly obvious: the “one-size-fits-all” foundation model strategy is dead. We are now living in a polarised ecosystem. On one side, you have cloud-tethered behemoths like Moonshot’s 2.8-trillion parameter Kimi K3; on the other, hyper-quantized reasoning engines like PrismML’s sub-4GB Bonsai 27B running completely offline on consumer hardware.

For solo developers and small engineering teams, this polarisation introduces a serious architectural headache. If you want to use a massive frontier model for complex data extraction, a mid-sized open-weight model for code generation, and a local model for fast classification on the same user workflow, your application code quickly devolves into a fragile mess of vendor SDKs, varying error schemas, and fragmented API keys.

Hardcoding vendor clients directly into your business logic is officially a production anti-pattern.

The solution isn’t to pick a single provider and compromise on capability or cost. An AI Gateway is a self-hosted middleware layer that exposes one OpenAI-compatible endpoint and routes each request to the right underlying model, decoupling your application code from individual vendor SDKs, credentials, and error formats. Here is how to spin up a unified infrastructure layer in less than 10 minutes.

Why Solo Developers Need a Gateway

Many developers assume that API gateways are enterprise bloat—middleware designed strictly for compliance, auditing, and corporate governance teams. In reality, a gateway is the ultimate force multiplier for a solo builder. It eliminates Integration Debt.

When you write code that talks directly to an external AI vendor, you aren’t just making an API call. You are committing to maintaining that vendor’s error-handling format, rate-limiting limits, and network timeout behaviours. If a cheaper, faster alternative drops next week, swapping models means tracking down string literals throughout your codebase, rewriting test suites, and refactoring initialisation logic.

An abstract gateway layer changes the equation completely:

• Unified Interface: Your application code speaks to exactly one local endpoint using a standard schema.
• Zero-Code Model Swaps: Want to test a new model weights release? Update a single line in a YAML configuration file. The application code remains completely untouched.
• Production Resilience for Free: If a primary cloud provider suffers an outage or returns a RateLimitError, the gateway intercepts the failure and instantly diverts the request to a backup provider before your user ever sees an error screen.

Technical Guide: The 10-Minute Gateway Setup

LiteLLM Proxy is an open-source gateway server that translates a single OpenAI-compatible API into vendor-specific calls for Anthropic, OpenAI, and self-hosted models served through Ollama, packaged as one lightweight Docker container.

If you haven’t containerised anything before, a Docker container is just a self-contained bundle of an application plus everything it needs to run (runtime, libraries, config), so it behaves identically on your laptop and in production. Docker Compose is the tool that starts several containers together from one YAML file, which is exactly what we’re using it for here — a one-file, one-command gateway. I cover the fundamentals in more depth in What is Docker?, including images versus containers and the Dockerfile basics; this post assumes that background and jumps straight to the gateway pattern.

Prerequisites: Docker, Python, and API Keys You Need

Before you start, make sure you have:

• Docker Desktop (or Docker Engine) 24+ with Compose v2 — check with docker compose version. Compose ships bundled with Docker Desktop; on Linux you may need the separate docker-compose-plugin package.
• Python 3.10+ for the client code, plus pip to install the openai SDK.
• API keys for whichever cloud providers you’re routing to — an ANTHROPIC_API_KEY and/or OPENAI_API_KEY. You only need the keys for the providers you actually list in config.yaml.
• Ollama installed locally if you want the local-inference fallback. Ollama is a small daemon that downloads open-weight models and serves them over an OpenAI-compatible local HTTP API on port 11434 — it’s what turns “a model on Hugging Face” into “a model you can curl”. Install it from ollama.com/download (or brew install ollama/tap/ollama on macOS), then pull the model ahead of time: ollama pull qwen2.5-coder:7b. I cover installing and calling Ollama end-to-end in DeepSeek R1 With Ollama. Skip this bullet entirely if you’re only routing to cloud providers.

1. The Gateway Topography (config.yaml)

The config.yaml file declaratively maps abstract internal aliases (like smart-reasoner or fast-code) to actual underlying infrastructure targets, completely hiding API locations, parameters, and credentials from your application.

model_list:
  - model_name: smart-reasoner
    litellm_params:
      model: anthropic/claude-3-5-sonnet-20241022
      api_key: "os.environ/ANTHROPIC_API_KEY"
      
  - model_name: fast-code
    litellm_params:
      model: openai/gpt-4o-mini
      api_key: "os.environ/OPENAI_API_KEY"

  - model_name: local-inference
    litellm_params:
      model: ollama/qwen2.5-coder:7b
      api_base: "http://host.docker.internal:11434" # Bridges container to a host-managed Ollama daemon

router_settings:
  routing_strategy: latency-based-routing
  enable_fallbacks: true

routing_strategy and enable_fallbacks are LiteLLM router settings: the first picks the fastest-responding deployment for a given alias, the second lets you chain a backup model behind it. Both live under router_settings regardless of which routing strategy you pick.

2. The Infrastructure Layer (docker-compose.yml)

Because it adheres to standard container patterns, you can drop the proxy straight into your existing Docker environment alongside your web servers and databases.

# No top-level "version:" key — Compose v2 ignores it and treats it as obsolete.
services:
  litellm-gateway:
    image: ghcr.io/berriai/litellm:main-stable
    container_name: litellm-proxy
    ports:
      - "4000:4000"
    volumes:
      - ./config.yaml:/app/config.yaml
    command: ["--config", "/app/config.yaml"]
    environment:
      - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
      - OPENAI_API_KEY=${OPENAI_API_KEY}
    restart: unless-stopped

3. Launching and Verifying the Gateway

Before writing any application code, get the proxy running and confirm it actually answers. Create a .env file alongside docker-compose.yml with your keys — Compose reads it automatically:

# .env
ANTHROPIC_API_KEY=sk-ant-...
OPENAI_API_KEY=sk-...

Then start the stack and check its health:

docker compose up -d
docker compose logs -f litellm-gateway   # watch it boot, Ctrl+C to detach

# once it's up, confirm the gateway answers on its OpenAI-compatible route
curl http://localhost:4000/v1/models

A working gateway returns a JSON list containing smart-reasoner, fast-code, and local-inference (if you configured Ollama). If curl hangs or refuses the connection, check docker compose logs first — a missing API key or an Ollama daemon that isn’t running are the two most common causes.

4. Clean Application Integration (app.py)

Because LiteLLM perfectly replicates the standard OpenAI JSON wire format, you don’t need fancy, niche libraries. You can use the official openai Python SDK or any basic HTTP client. Install it first:

# requirements.txt
openai>=1.40.0

pip install -r requirements.txt

"""Minimal client for the local LiteLLM gateway.

Run the gateway first (docker compose up -d), then:
    python app.py
"""
import logging
import os

import openai

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

# The gateway URL is configurable via an environment variable, defaulting to
# the port used in docker-compose.yml above. This avoids hardcoding
# localhost:4000 everywhere and makes the client trivial to point at a
# staging gateway, or a disposable one in a test suite, without editing code.
GATEWAY_BASE_URL = os.environ.get("LITELLM_BASE_URL", "http://localhost:4000/v1")

# Bind your app client to the local gateway network node.
# The API key is a placeholder: LiteLLM only enforces it if you configure
# virtual keys (see "Enforce Virtual Keys and Financial Budgets" below) —
# once you do, this must become a real generated key or requests are rejected.
client = openai.OpenAI(
    base_url=GATEWAY_BASE_URL,
    api_key=os.environ.get("LITELLM_VIRTUAL_KEY", "sk-not-required-for-local-defaults"),
)


def execute_agent_pipeline(prompt: str, model: str = "smart-reasoner") -> str:
    """Send a prompt through the gateway using a functional model alias.

    Args:
        prompt: The user-facing instruction to send to the model.
        model: The alias defined in config.yaml (smart-reasoner, fast-code,
            local-inference) — not a real provider model ID. The gateway
            resolves the alias to an actual model at request time, so this
            function never needs to know which vendor is behind it.

    Returns:
        The model's text response, or a readable fallback message if the
        gateway or every configured upstream provider is unavailable.
    """
    try:
        response = client.chat.completions.create(
            model=model,
            messages=[
                {"role": "system", "content": "You are a specialised code optimisation agent."},
                {"role": "user", "content": prompt},
            ],
            temperature=0.1,
        )
        return response.choices[0].message.content or ""
    except openai.RateLimitError:
        # All configured providers for this alias are rate-limited or over
        # budget (see the virtual-key budgets upgrade below).
        logger.warning("Rate limited on %s; consider a virtual key budget or a fallback alias.", model)
        return "Rate limit hit across configured providers."
    except openai.APIConnectionError:
        # Could not reach the gateway at all — usually means the
        # litellm-gateway container isn't running.
        logger.error("Could not reach the LiteLLM gateway at %s", client.base_url)
        return "Infrastructural connection failure across available providers."
    except openai.APIStatusError as exc:
        # The gateway responded but with a non-2xx status — e.g. the
        # upstream provider rejected the request (bad API key, invalid
        # model alias, content policy violation).
        logger.error("Gateway returned an error status: %s", exc.status_code)
        return f"Upstream provider error ({exc.status_code})."


if __name__ == "__main__":
    result = execute_agent_pipeline("Refactor this function to use list comprehensions.")
    print(result)

I have packaged this exact code as a tested, runnable example — including a smoke test that starts a real LiteLLM proxy with mocked model responses, so it runs without API keys or Docker — in the multi_llm folder of my coding examples collection. If you hit an issue copying the snippets above, that folder is the one to diff against.

Scaling the Gateway: Observability, Budgets, and Tracing

Once this baseline abstraction is running in production, you can layer on powerful optimizations incrementally without rewriting a single line of your business applications.

1. Implement a Persistent Observability Cache

By default, the proxy holds operational state in-memory. By linking a standard postgres:alpine instance to the gateway configuration via a DATABASE_URL string, you unlock persistent tracking. You get a central database of exact token spends, latency profiles, and call counts across every model you use—all under your absolute physical control. Add it as a second service in docker-compose.yml:

services:
  db:
    image: postgres:alpine
    environment:
      - POSTGRES_DB=litellm
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
    volumes:
      - litellm_data:/var/lib/postgresql/data

  litellm-gateway:
    # ...existing config from above...
    environment:
      - DATABASE_URL=postgresql://postgres:${POSTGRES_PASSWORD}@db:5432/litellm
    depends_on:
      - db

volumes:
  litellm_data:

2. Enforce Virtual Keys and Financial Budgets

If you run automated testing suites on Git branch commits or build autonomous agents that execute multi-step loops, a runaway script can rack up massive bills in minutes. A Virtual Key is a per-consumer credential the gateway issues and tracks independently of your real provider API keys, letting you impose hard ceilings (e.g., maximum $5 total spend over a rolling 30 days) right at the endpoint layer. If the agent goes haywire, the gateway kills the key, protecting your wallet. Generate one against your running proxy (requires the Postgres backend above):

curl -X POST http://localhost:4000/key/generate \
  -H "Authorization: Bearer sk-master-key-from-config" \
  -H "Content-Type: application/json" \
  -d '{"max_budget": 5, "budget_duration": "30d", "models": ["fast-code"]}'

3. Stream Traces to Langfuse via Callbacks

Langfuse is an open-source LLM observability platform that ingests traces, token counts, and latency data through webhook-style callbacks. Instead of scattering telemetry hooks inside your application, let the gateway pipe execution logs out-of-band. Adding a success_callback/failure_callback pair to your config.yaml can automatically dispatch trace states and system metadata directly to Langfuse or custom logging frameworks, keeping your execution pathways lean, readable, and lightning fast:

litellm_settings:
  success_callback: ["langfuse"]
  failure_callback: ["langfuse"]

Final Thoughts: A Vendor-Agnostic AI Gateway for Solo Developers

The engineering goal for a solo developer or small team shouldn’t be to find the single “best” model; it should be to maximise engineering velocity and resilience while minimising vendor dependency. By investing ten minutes into setting up an architectural abstraction layer today, you give yourself the freedom to navigate the rapidly shifting AI landscape entirely on your own terms.

How are you managing model complexity in your current pipelines? Are you leaning toward managed aggregators or sticking to pure self-hosted infrastructure layouts? Let me know your thoughts.

References

• LiteLLM Proxy Docker Quick Start
• LiteLLM Routing & Load Balancing
• LiteLLM Virtual Keys
• LiteLLM Langfuse Integration
• Ollama OpenAI Compatibility
• Moonshot AI’s Kimi K3 (VentureBeat)
• PrismML Bonsai 27B announcement