Production Hardening: Idempotency, Run Isolation, and Crash Safety

So far in this series, my system works. But production systems are not tested by success. They are tested by failure, and I would rather find that out on my own terminal than at 2am from a Slack message.

So let me ask my own setup some uncomfortable questions:

• What if the container crashes mid-run?
• What if Slack sends the same approval twice?
• What if I click Approve twice by accident?
• What if I restart Docker while the graph is sitting on an interrupt?
• What if two runs share the same output folder?
• What if the crash lands exactly between a side effect and the checkpoint that was supposed to record it?

Right now, my system might survive these by luck. After this post, it handles them deliberately — and I will be honest about where “deliberately” still has edges.

Stable Identity: Job, Run, and Thread Are Not the Same Thing

I started this hardening pass by hashing a snippet of the newsletter intro into a thread_id. It worked, right up until I thought about it properly:

import hashlib

def generate_thread_id(intro: str) -> str:
    base = intro[:50]
    digest = hashlib.sha256(base.encode()).hexdigest()[:8]
    return f"newsletter-{digest}"

Three problems, in ascending order of embarrassment. First, I only hash the first 50 characters, so two newsletters that happen to open with the same boilerplate greeting collide deterministically — not a hypothetical, just a matter of time. Second, eight hex characters is 32 bits of identifier space, which is thin for very little reason; there is no cost to using 16. Third, and the one that actually matters: identical content and identical execution are not the same concept. If I intentionally rerun a newsletter after tweaking the prompt, should that resume the old workflow, silently overwrite it, or start a fresh attempt? A thread_id built purely from content cannot answer that question, because it was never designed to.

So I split identity into three layers instead of one:

job_id      = identity of the logical newsletter (this Friday's issue)
run_id      = identity of one particular execution attempt
thread_id   = the LangGraph checkpoint stream for that execution

from datetime import date
import hashlib
import re


def slugify(value: str) -> str:
    value = value.lower().strip()
    return re.sub(r"[^a-z0-9]+", "-", value).strip("-")


def make_job_id(newsletter_date: date, title: str) -> str:
    return f"newsletter-{newsletter_date.isoformat()}-{slugify(title)}"


def make_thread_id(job_id: str, source_text: str, workflow_version: str) -> str:
    payload = "\0".join([job_id, source_text, workflow_version])
    digest = hashlib.sha256(payload.encode("utf-8")).hexdigest()[:16]
    return f"{job_id}-{digest}"

job_id is what I look up in logs and dashboards. thread_id is what the checkpointer actually keys on — and because it hashes the full source text plus a workflow version rather than the first 50 characters of an intro, a rerun after a prompt change gets a new, distinguishable thread instead of quietly colliding with the old one. If I want explicit reruns rather than content-derived ones, I append an attempt suffix: newsletter-2026-08-07-ai-signals-a1, -a2, and so on. LangGraph’s checkpointer uses the exact thread_id as its lookup key — lose it or collide it, and you lose the ability to resume that run at all, so this is not a place to be casual.

Durable Checkpoints: SQLite Today, PostgreSQL at Scale

I am using a SQLite checkpointer, a stable thread_id, and a persistent checkpoints.db file. I need to be more precise than I was tempted to be about what that buys me.

LangGraph’s own documentation describes SqliteSaver as ideal for experimentation and local workflows, and points to PostgresSaver as the option for production. That is not a knock on SQLite — for this single-host workflow, with one orchestration process and low write concurrency, SQLite gives me durable recovery across container restarts, and there is no reason to reach for Postgres before I need it. It stops being the right tool once I add multiple replicas, several workers writing concurrently, or a network-attached filesystem. So: for this workflow, SQLite is enough. For the next one with concurrent workers, I would move the checkpointer to PostgreSQL rather than assume SQLite scales with me.

There is also a quieter failure mode worth naming: a container being recreated is not the same operation as a container being restarted. Without a persistent volume, checkpoints.db disappears the moment Docker replaces the container, and no amount of correct thread_id logic saves you from that. I mount it explicitly:

services:
  app:
    volumes:
      - checkpoints:/app/data
      - artifacts:/app/out

volumes:
  checkpoints:
  artifacts:

That one YAML block is the difference between “the crash test passed” and “the crash test passed because I happened not to recreate the container.”

Per-Run Artifact Isolation, and Why Isolation Is Not Atomicity

Before this pass, every run wrote to the same shared place:

out/
  newsletter.md
  report.json

Fine for a tutorial, dangerous in production: a second run in flight silently overwrites the first one’s files while they are still being written. I isolate output per thread_id instead — the same identifier LangGraph already uses to track a run’s checkpointed state:

BASE_ARTIFACT_DIR = Path("out")

def get_run_dir(thread_id: str) -> Path:
    run_dir = BASE_ARTIFACT_DIR / thread_id
    run_dir.mkdir(parents=True, exist_ok=True)
    return run_dir

out/
  newsletter-2026-08-07-ai-signals-a1b2c3d4e5f6a7b8/
    newsletter.md
    report.json
    report.md

Two correctly identified runs no longer write to the same artifact paths — I will not claim “never,” because a thread_id collision or a sanitisation bug could still create overlap, and “never” is exactly the kind of word that ages badly in a production post.

But isolation only stops runs from stepping on each other. It does nothing about a single run stepping on itself mid-write. This:

path.write_text(large_report)

can be interrupted by a crash halfway through, leaving a report.json that exists, sits on disk, and cannot be parsed by anything downstream. Isolation and atomicity solve different problems, so I treat them as two separate fixes. The atomic version writes to a temp file in the same directory and replaces the target only once the write is complete and flushed to disk:

import json
import os
import tempfile
from pathlib import Path
from typing import Any


def atomic_write_json(path: Path, value: Any) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    fd, temporary_name = tempfile.mkstemp(
        dir=path.parent,
        prefix=f".{path.name}.",
        suffix=".tmp",
    )
    temporary_path = Path(temporary_name)
    try:
        with os.fdopen(fd, "w", encoding="utf-8") as file:
            json.dump(value, file, indent=2)
            file.flush()
            os.fsync(file.fileno())
        temporary_path.replace(path)
    except Exception:
        temporary_path.unlink(missing_ok=True)
        raise


def atomic_write_text(path: Path, content: str) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    temporary_path = path.with_suffix(path.suffix + ".tmp")
    temporary_path.write_text(content, encoding="utf-8")
    temporary_path.replace(path)

Path.replace is a single filesystem rename, so the reader of report.json only ever sees the complete old file or the complete new one — never something in between. Every file write in the graph now goes through one of these two helpers.

Interrupt Replay: Why “Repeat” Is the Default, Not the Exception

I need idempotency in the first place because of how LangGraph resumes a paused run. When a graph hits an interrupt() and later resumes, it does not continue from the exact line where it stopped. LangGraph’s documentation on durable execution is explicit: the workflow replays from the start of the node that was interrupted, re-running any code that sat before the interrupt call. For a StateGraph, the starting point on resume is always the beginning of the node — not a saved instruction pointer inside it.

That single fact is the reason this entire post exists. Any file write, Slack call, or finalisation step placed before an interrupt in a node is not “occasionally” re-run on resume — it is re-run by design, every time. The documentation’s own advice follows directly from this: wrap side effects in tasks, or make them idempotent, or both.

Idempotent Finalisation: Three Levels of Protection

Here is the finalisation node as I first wrote it:

def node_finalize_report(state: EditorialState) -> dict:
    if state.get("finalized"):
        return {}

    write_final_report(state)
    return {"finalized": True}

That is a graph-state guard, and it is genuinely useful — but on its own it is not enough to call this idempotent. There is a gap between the report being written and finalized: True being safely checkpointed, and a crash landing in that exact gap is not a contrived edge case:

Write report.md successfully
        ↓
Container crashes
        ↓
LangGraph never checkpoints finalized=True
        ↓
Node runs again on resume
        ↓
report.md is written again, or a notification is sent twice

A boolean in graph state protects against ordinary repeated execution once the state update has been persisted. It cannot, by itself, close the “side effect succeeded, checkpoint failed” window. So I think about idempotency in three levels, and I only reach for the third when the second genuinely is not enough.

Level 1 — the graph-state guard. This is what I had. It stops a resumed node from redoing work once its own finalized: True is safely on disk. Cheap, necessary, insufficient alone.

Level 2 — an idempotency key stored beside the side effect. For an operation against an external system — Slack, email, a database — I give the operation its own key and claim it atomically before acting:

idempotency_key = f"{thread_id}:final-report"

CREATE TABLE completed_operations (
    idempotency_key TEXT PRIMARY KEY,
    completed_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);

def claim_operation(conn, key: str) -> bool:
    cursor = conn.execute(
        "INSERT OR IGNORE INTO completed_operations(idempotency_key) VALUES (?)",
        (key,),
    )
    conn.commit()
    return cursor.rowcount == 1


def finalize_report(state: EditorialState) -> dict:
    thread_id = state["thread_id"]
    operation_key = f"{thread_id}:final-report"

    if not claim_operation(db, operation_key):
        return {"finalized": True}

    write_final_report(state)
    return {"finalized": True}

The exact ordering here still depends on whether the downstream system supports transactions or has its own idempotency keys — I am not pretending this is the only correct shape — but the underlying idea is the one worth keeping: idempotency belongs to the operation, not merely to the graph state.

Level 3 — the transactional outbox pattern, briefly. For the deepest protection, I do not send Slack or email directly from the node at all. Finalisation writes the completed report and an “outbox event” in a single database transaction. A separate worker reads unsent outbox events, sends the message, marks the event delivered, and reuses the same event ID on any retry. I have not implemented the full pattern here — it is more machinery than a personal newsletter pipeline currently justifies — but it is how you decouple a graph checkpoint from an unreliable external network call, and it is worth knowing the name for when the workflow outgrows Level 2.

The mental model I use for all three levels: idempotency is an elevator call button, not a doorbell. Press a doorbell three times and you might summon three answers. Press an already-lit elevator button three times and nothing extra happens — the elevator was already coming. That is the behaviour I want from a Slack approval click, a webhook retry, or a resumed graph: pressing it again should not call a second elevator.

Duplicate Deliveries and the Double-Click Race

The naive guard —

if not finalized:
    finalize()

— is vulnerable to more than sequential retries. Two requests can arrive close enough together that both read finalized=False before either has recorded completion:

Request A reads finalized=False
Request B reads finalized=False
Request A resumes graph
Request B resumes graph

Both passed the check before either committed. That is a check-then-act race, and no amount of careful sequencing inside one request fixes a problem that happens between two requests. The endpoint’s finalized check is a convenience, not a guarantee — the authoritative duplicate protection has to be an atomic insert or a unique constraint in durable storage:

INSERT INTO approval_actions (thread_id, action_id, decision)
VALUES (?, ?, ?)
ON CONFLICT(action_id) DO NOTHING;

That matters especially at the Slack boundary, because a webhook payload is not a full EditorialState handed to you on a plate — my first draft of this endpoint pretended it was:

@app.post("/slack/actions")
def slack_actions(state: EditorialState):
    if state.get("finalized"):
        return {"ok": True, "message": "Run already finalized."}
    ...

A real Slack action arrives as a form-encoded payload that needs its signature verified, its action identifier and thread ID extracted, and the graph state loaded through the checkpointer before anything else happens:

from fastapi import Request
from langgraph.types import Command


@app.post("/slack/actions")
async def slack_actions(request: Request):
    payload = await parse_and_verify_slack_request(request)
    thread_id = payload["actions"][0]["value"]

    config = {"configurable": {"thread_id": thread_id}}
    snapshot = graph.get_state(config)

    if snapshot.values.get("finalized"):
        return {"ok": True, "message": "Run already finalized."}

    decision = payload["actions"][0]["action_id"] == "approve"
    graph.invoke(Command(resume=decision), config=config, durability="sync")
    return {"ok": True}

That version verifies the Slack signature, keys everything off thread_id rather than a full state blob, reads current state through the checkpointer instead of trusting the request body, and resumes with Command(resume=...) against the same thread — which is the documented way to feed a decision back into a paused graph. Duplicate delivery, or a duplicate click, now lands on a snapshot that already says finalized: True and does nothing.

An Explicit Run Lifecycle

I had three outcome labels: approved, rejected, max_revisions_exceeded. Those describe a business decision, but they say nothing about whether the operation itself succeeded. A human can approve a report that then fails to finalise — and a single flattened status field hides exactly that distinction, which is the one I most want visible in a debugging session.

So I separate run status, decision, and termination reason instead of collapsing them into one field:

from typing import Literal, TypedDict

RunStatus = Literal["pending", "waiting_for_approval", "finalizing", "completed", "failed"]
Decision = Literal["approved", "rejected", "not_decided"]
TerminationReason = Literal[
    "approved", "rejected", "max_revisions_exceeded", "cancelled", "execution_error"
]


class EditorialState(TypedDict, total=False):
    status: RunStatus
    decision: Decision
    termination_reason: TerminationReason
    finalized: bool
    error_message: str

Now a run can legitimately be status="failed", decision="approved", termination_reason="execution_error" — the human said yes, finalisation blew up, and that is a materially different situation from a straightforward rejection. A single status="approved" field would have quietly hidden it.

Crash Recovery: Testing a Container Kill, Systematically

I actually ran this rather than just reasoning about it:

1. Start a run.
2. Let it reach the Slack interrupt.
3. Kill the container.
4. Restart Docker.
5. Click Approve in Slack.

With a SQLite checkpointer, a stable thread_id, a persistent volume, and durability="sync" on the finalisation call, LangGraph reads the last saved checkpoint and resumes from there instead of starting over. That is what I mean by production-minded — not “it probably won’t crash,” but “a crash becomes a tested recovery path rather than an unrecoverable event.” I am deliberately not calling it “it does not matter if it crashes,” because a crash landing inside a non-transactional side effect — the exact gap discussed above — can still matter, which is the whole reason for the idempotency-key work.

How durable that recovery is depends on the durability mode the graph runs with. LangGraph documents three:

For finalisation I would rather lose a little speed than lose a report, so this is the one place I invoke the graph with synchronous durability explicitly:

graph.invoke(Command(resume=decision), config=config, durability="sync")

I want to be precise about what that buys me, because it would be easy to oversell: sync guarantees the checkpoint write finishes before execution continues, which shrinks the window in which graph state can be lost. It does not make an external API call and a checkpoint write one atomic transaction — a Slack call can still succeed while the checkpoint that would have recorded it fails, entirely independently of which durability mode I chose. That is precisely why the external operation needs its own idempotency protection from the section above, rather than borrowing safety from the durability mode.

One test is a start, not a suite. I now check a matrix of failure points rather than a single happy path through the crash:

The shell version of the first row is short enough to run before every deploy:

docker compose up -d
curl -X POST http://localhost:8000/runs \
  -H 'Content-Type: application/json' \
  -d '{"title":"Friday AI Signals"}'
docker compose kill app
docker compose up -d app

Then I inspect the resumed thread with graph.get_state(config) and check the artifact directory landed exactly once, with exactly one complete report.json in it.

Production Hardening Checklist for LangGraph Workflows

1. Assign each logical job a stable job_id.
2. Assign each execution a unique or reproducible thread_id, derived from stable input plus a workflow version — never from a random value.
3. Store checkpoints and artifacts on persistent storage, mounted explicitly as a volume.
4. Isolate every run’s artifacts in their own folder, keyed by that thread_id.
5. Write final artifacts atomically — temp file, then replace, never a direct write to the final path.
6. Make external side effects idempotent with an operation-level key, not just a finalized flag in graph state.
7. Protect webhook handling with atomic deduplication — a unique constraint, not a check-then-act read.
8. Distinguish run status, human decision, and termination reason instead of one flattened outcome label.
9. Test crashes at several execution boundaries, not just one happy-path kill.
10. Test concurrent runs and duplicate approval delivery, not only sequential retries.

If you implement these ten, you are no longer experimenting with an AI workflow. You are operating one.

This pattern is not specific to newsletters — it applies just as directly to AI code generation workflows, customer support triage agents, document summarisation pipelines, security scanning workflows, and data enrichment pipelines. Any workflow that can be interrupted needs the same guarantees.

If there is one idea I would want to stick from this whole post, it is this: I stopped promising myself “exactly once” execution, because in a distributed system that promise is usually a polite fiction. The realistic goal is at-least-once delivery with idempotent processing and observable outcomes — and that is a much sturdier thing to build on than a boolean flag ever was.

What This Hardening Pass Actually Buys You

Hardening a workflow like this is not exciting work, and that is rather the point. My system now has a worker, a supervisor, a retry loop, human approval, interrupt-and-resume, MCP tool isolation, per-run artifact folders with atomic writes, idempotent finalisation backed by an operation key, and a crash-test matrix instead of a single manual poke. It is stable enough now that the next problem is not “will it survive,” but “can I actually see what it is doing.” That is what I want to tackle next: structured logs, correlation IDs, execution timing, and a run summary that tells me what happened without me having to guess.

References

• LangChain — LangGraph Persistence (checkpointers, threads, SqliteSaver, PostgresSaver)
• LangChain — LangGraph Durable Execution (idempotency, tasks, replay on resume, durability modes)
• LangChain — LangGraph Interrupts (human-in-the-loop pause/resume, Command(resume=...))