Production Hardening: Idempotency, Run Isolation, and Crash Safety

So far in this series, my system works. But production systems are not tested by success. They are tested by failure, and I would rather find that out on my own terminal than at 2am from a Slack message.

So let me ask my own setup some uncomfortable questions:

• What if the container crashes mid-run?
• What if Slack sends the same approval twice?
• What if I click Approve twice by accident?
• What if I restart Docker while the graph is sitting on an interrupt?
• What if two runs share the same output folder?

Right now, my system might survive these by luck. After this post, it handles them deliberately.

Per-Run Artifact Isolation

Right now I write every run’s output to the same place:

out/
  newsletter.md
  report.json

That is fine for a tutorial. It is dangerous in production, because a second run in flight will silently overwrite the first one’s files while it is still writing them.

Instead, I isolate output per thread_id — the same identifier LangGraph (a graph-based orchestration framework for stateful, multi-step LLM workflows) already uses to track a run’s checkpointed state, per LangGraph’s persistence docs:

out/
  newsletter-001/
    newsletter.md
    report.json
  newsletter-002/
    ...

In app/server.py, I replace the single shared directory:

ARTIFACT_DIR = Path("out")

with a per-run helper:

BASE_ARTIFACT_DIR = Path("out")

def get_run_dir(thread_id: str) -> Path:
    run_dir = BASE_ARTIFACT_DIR / thread_id
    run_dir.mkdir(parents=True, exist_ok=True)
    return run_dir

Then every file write goes through it:

run_dir = get_run_dir(thread_id)
(run_dir / "newsletter.md").write_text(...)
(run_dir / "report.json").write_text(...)
(run_dir / "report.md").write_text(...)

Every run now has its own folder. Two runs in flight can no longer step on each other’s files.

Idempotency: Why Repeating a Step Must Be Safe

Idempotency is a property of an operation where repeating it produces the same outcome as running it once — a Slack retry, a double-click, a resumed graph, none of them should be able to double-write a report or double-post an approval message.

I need it here specifically because of how LangGraph resumes a paused run. When a graph hits an interrupt() and later resumes, LangGraph does not continue from the exact line where it stopped — it re-runs the whole node from the start, replaying any code that sat before the interrupt. LangGraph’s own documentation on durable execution is explicit about this: side effects placed before an interrupt need to be idempotent, or wrapped in a task, precisely because they can execute again on resume. The same page recommends using idempotency keys or checking for an existing result before writing, to avoid duplicating work when a step is retried.

I apply that to three places: Slack approval resume, finalisation, and file writes. The cheapest fix is a finalised flag in state:

class EditorialState(TypedDict, total=False):
    ...
    finalized: bool

checked at the top of the finalisation node:

def node_finalize_report(state: EditorialState) -> EditorialState:
    if state.get("finalized"):
        return state

    state["finalized"] = True
    return state

Now if Slack triggers the resume twice, the node notices it already finished and returns without regenerating anything or re-running a side effect.

Deterministic Thread IDs

I never rely on a random ID in production, because LangGraph’s checkpointer looks up a run’s saved state by exact thread_id — lose the ID, and you lose the ability to resume that run at all. Instead, I derive it from something stable about the run itself: the newsletter date, the slug, a short content hash.

import hashlib

def generate_thread_id(intro: str) -> str:
    base = intro[:50]
    digest = hashlib.sha256(base.encode()).hexdigest()[:8]
    return f"newsletter-{digest}"

The same intro now always produces the same thread_id, which means a restart or a resume can always find its way back to the correct checkpoint.

Crash Recovery: Testing a Container Kill Mid-Interrupt

I actually ran this test rather than just reasoning about it:

1. Start a run.
2. Let it reach the Slack interrupt.
3. Kill the container.
4. Restart Docker.
5. Click Approve in Slack.

Because I am using a SQLite checkpointer, a stable thread_id, and a persistent checkpoints.db file, LangGraph reads the last saved checkpoint and resumes from there rather than starting over. That is what production-level behaviour actually looks like — not “it probably won’t crash,” but “it does not matter if it crashes.”

How durable that recovery is depends on the durability mode the graph runs with. LangGraph documents three:

For a finalisation step I would rather lose a little speed than lose a report, so this is the one place in the graph where I use sync.

Prevent Double Resume

The /slack/actions endpoint gets the same finalised check as the node itself:

@app.post("/slack/actions")
def slack_actions(state: EditorialState):
    if state.get("finalized"):
        return {"ok": True, "message": "Run already finalized."}
    ...

That single guard makes clicking the Slack button twice, or Slack retrying a webhook delivery, harmless instead of destructive.

Failure Classification

A run that finishes should say clearly what happened to it, not just that it finished:

state["report"]["status"] = (
    "approved"
    if state.get("human_approved")
    else "rejected"
)

I extend this to three explicit states: approved, rejected, and max_revisions_exceeded. A report that silently says nothing about why a run stopped is a debugging session waiting to happen.

Newsletter Artifacts: Shared File vs. Per-Run Folder

Before, every run wrote to the same shared file:

newsletter.md (shared)

After isolating per-run artifacts with a deterministic thread ID, the same output looks like this:

out/
  newsletter-9f3a2c1b/
    newsletter.md
    report.json
    report.md

Now I can rerun a newsletter safely, compare two runs side by side, archive by date, and audit exactly what happened on a given day — none of which was possible when every run fought over the same file.

Idempotency Mental Model: The Elevator Button Analogy

Idempotency is an elevator call button, not a doorbell. Press a doorbell three times and you might summon three answers. Press an already-lit elevator button three times and nothing extra happens — the elevator was already coming. That is the behaviour I want from a Slack approval click, a webhook retry, or a resumed graph: pressing it again should not call a second elevator.

Production Hardening Checklist for LangGraph Workflows

1. Give every run a deterministic thread_id, derived from stable input, not a random value.
2. Isolate every run’s artifacts in their own folder, keyed by that thread_id.
3. Make every external side effect idempotent — Slack replies, file writes, finalisation — by checking a finalized flag before acting.
4. Actually test a crash: kill the process mid-interrupt, restart it, and resume, rather than assuming it will work.
5. Classify the outcome explicitly (approved / rejected / max_revisions_exceeded) so a stopped run never leaves you guessing why.

If you implement these five, you are no longer experimenting with an AI workflow. You are operating one.

This pattern is not specific to newsletters — it applies just as directly to AI code generation workflows, customer support triage agents, document summarisation pipelines, security scanning workflows, and data enrichment pipelines. Any workflow that can be interrupted needs the same guarantees.

What This Hardening Pass Actually Buys You

Hardening a workflow like this is not exciting work, and that is rather the point. My system now has a worker, a supervisor, a retry loop, human approval, interrupt-and-resume, MCP tool isolation, per-run artifact folders, idempotent finalisation, and crash safety. It is stable enough now that the next problem is not “will it survive,” but “can I actually see what it is doing.” That is what I want to tackle next: structured logs, correlation IDs, execution timing, and a run summary that tells me what happened without me having to guess.

References

• LangChain — LangGraph Persistence (checkpointers, threads, SqliteSaver)
• LangChain — LangGraph Durable Execution (idempotency, tasks, replay on resume, durability modes)
• LangChain — LangGraph Interrupts (human-in-the-loop pause/resume)