Last week, a founder woke up to find their MRR had collapsed to $38. No customers had canceled. No refunds had been issued. While they slept, a cron job written by GPT-5.6-Sol (Codex) had canceled every active Stripe subscription their business had. All of them. In seven seconds.
The tweet went viral. The narrative: "GPT 5.6 cannot be trusted. Fable 5 has never done this to me."
That framing is understandable. It's also wrong — and believing it will get you burned by whichever model you switch to next.
What Actually Happened
The bug itself is a classic one: a cron job that treated an empty deletion queue as a valid input to process everything. The code checked for items to delete, found none, and instead of aborting — it ran the deletion logic on the full subscription set. Empty queue = "do all."
This is not an AI hallucination. It's not a model alignment failure. It's a logic error that human developers have been writing since the 1980s. The fact that the code was generated by an AI model is almost irrelevant to the root cause.
What is relevant: that code had direct, unguarded access to a live production Stripe API key, was scheduled to run autonomously, and had no human checkpoint between "AI generated this" and "this ran on your entire customer base."
The Actual Failure Mode: No Safety Rails on Destructive Operations
Every agentic AI pipeline that touches production data needs to answer three questions:
- What operations are irreversible? Canceling subscriptions, deleting records, sending emails — these cannot be undone. They need a different class of treatment than read operations.
- What credentials does the AI agent hold? A Stripe API key that can cancel subscriptions should never be handed directly to an autonomous agent. Scoped, restricted keys exist for exactly this reason.
- What's the human checkpoint before destructive actions execute? "The code looked right when I reviewed it" is not a checkpoint. A checkpoint is: show me the list of records that will be affected before you touch them, require my explicit approval for anything irreversible.
None of these were in place. That's the failure.
Three Rails That Would Have Prevented This
1. Dry-run gates with non-empty checks. Any agent running a deletion or cancellation loop should first log what it would do and halt if the target set is empty or unexpectedly large. Empty queue → abort, alert, never proceed. This is a two-line check that would have stopped this incident cold.
2. Scoped credentials, not production keys. AI agents should receive the minimum Stripe permissions needed for their task — ideally read-only by default, with write access requiring explicit escalation. Restricted API keys are a Stripe feature. Use them.
3. Human-in-the-loop for irreversible ops. Async approval for any operation that can't be undone. A Slack message: "I'm about to cancel 47 subscriptions. Confirm?" takes three seconds to respond to. It costs nothing. It makes autonomous agents safe to run while you sleep.
Why Model-Blaming is a Trap
The "trust Fable 5, not GPT-5.6" framing creates a false sense of security. Every current AI model can write a cron job with a logic error. The rate varies. The risk doesn't disappear. Architectural safety rails are model-agnostic — they protect you regardless of which model wrote the code, which model runs the agent, or which model you switch to next quarter.
Matt Shumer posted the same day that GPT-5.6-Sol had accidentally deleted almost all of his Mac's files. Different product, same pattern: an AI agent with unguarded access to destructive operations and no human checkpoint before execution.
The common thread isn't the model. It's the architecture.
The Production-Ready Agentic Pipeline
If you're running AI agents against production systems, the checklist is short:
- Classify every operation: read / write-reversible / write-irreversible
- Require explicit human approval for all write-irreversible operations
- Scope credentials to the minimum necessary permission level
- Gate deletion/cancellation loops on non-empty, size-bounded target sets
- Run in sandbox with production-mirrored data before touching live systems
- Log everything the agent did, in plain language, before it does it
None of this requires a specific model. All of it is required before you let any model run autonomously against systems that matter.
The founder's MRR is gone. The cron job ran as designed — except the design had no safety rails. That's the lesson. Build the rails before you run the agent, not after you wake up to an empty dashboard.
Originally published at vsebude.it