Production Hardening: Idempotency, Run Isolation, and Crash Safety

So far in this series, my system works. But production systems are not tested by success. They are tested by failure, and I would rather find that out on my own terminal than at 2am from a Slack message.

So let me ask my own setup some uncomfortable questions:

• What if the container crashes mid-run?
• What if Slack sends the same approval twice?
• What if I click Approve twice by accident?
• What if I restart Docker while the graph is sitting on an interrupt?
• What if two runs share the same output folder?
• What if the crash lands exactly between a side effect and the checkpoint that was supposed to record it?

Right now, my system might survive these by luck. After this post, it handles them deliberately — and I will be honest about where “deliberately” still has edges.

Stable Identity: Job, Run, and Thread Are Not the Same Thing

I started this hardening pass by hashing a snippet of the newsletter intro into a thread_id. It worked, right up until I thought about it properly:

import hashlib

def generate_thread_id(intro: str) -> str:
    base = intro[:50]
    digest = hashlib.sha256(base.encode()).hexdigest()[:8]
    return f"newsletter-{digest}"

Three problems, in ascending order of embarrassment. First, I only hash the first 50 characters, so two newsletters that happen to open with the same boilerplate greeting collide deterministically — not a hypothetical, just a matter of time. Second, eight hex characters is 32 bits of identifier space, which is thin for very little reason; there is no cost to using 16. Third, and the one that actually matters: identical content and identical execution are not the same concept. If I intentionally rerun a newsletter after tweaking the prompt, should that resume the old workflow, silently overwrite it, or start a fresh attempt? A thread_id built purely from content cannot answer that question, because it was never designed to.

So I split identity into three layers instead of one:

job_id      = identity of the logical newsletter (this Friday's issue)
run_id      = identity of one particular execution attempt
thread_id   = the LangGraph checkpoint stream for that execution

from datetime import date
import hashlib
import re

def slugify(value: str) -> str:
    value = value.lower().strip()
    return re.sub(r"[^a-z0-9]+", "-", value).strip("-")

def make_job_id(newsletter_date: date, title: str) -> str:
    return f"newsletter-{newsletter_date.isoformat()}-{slugify(title)}"

def make_thread_id(job_id: str, source_text: str, workflow_version: str) -> str:
    payload = "\0".join([job_id, source_text, workflow_version])
    digest = hashlib.sha256(payload.encode("utf-8")).hexdigest()[:16]
    return f"{job_id}-{digest}"

job_id is what I look up in logs and dashboards. thread_id is what the checkpointer actually keys on — and because it hashes the full source text plus a workflow version rather than the first 50 characters of an intro, a rerun after a prompt change gets a new, distinguishable thread instead of quietly colliding with the old one. If I want explicit reruns rather than content-derived ones, I append an attempt suffix: newsletter-2026-08-07-ai-signals-a1, -a2, and so on. LangGraph’s checkpointer uses the exact thread_id as its lookup key — lose it or collide it, and you lose the ability to resume that run at all, so this is not a place to be casual.