I tried to make AI charge a customer twice

I set out to make AI-written payment code charge a customer twice. It almost never did.

What it did instead was quieter, and in some ways worse. In one checkout, the code took the customer's money correctly and then told them their card had been declined. I then asked Claude Opus 5.5, twice, to review a checkout with exactly that bug. Both times it said the code was safe.

None of this touched real money. The customer, the card, the database and the payment provider are all simulated. I wrote the provider myself and called it PayCo.

If you have worked on payments, you know the bug I was hunting. A customer presses Pay. The charge goes through, but the reply is lost on the way back. The code times out, tries again, and the customer pays twice. The fix is an idempotency key, a unique ID sent with the charge, so that the provider returns the first result instead of charging again. Every model knows that. The details are where payment systems lose money: what happens when the server dies between two database writes, and what the customer sees while two requests for the same order run at once.

AI agents now write a lot of this code, and review a lot of it too. I build payment backends, so I started where I can tell right from wrong. Does their code hold up when replies are lost and the server crashes? And when a model calls someone else's code safe, can I believe it?

A payment system that fails on purpose

So I built DoubleCharge. Nothing in it is real: no bank, no card, no payment platform. PayCo, my provider, behaves the way Stripe's documentation says Stripe behaves with idempotency keys, timeouts and webhooks, and it can be told to misbehave on cue. Against a real provider you cannot kill the server at every call, or replay the exact same collision between two requests, or let anyone rerun your results for free.

Three frontier models, Claude Opus 5.5, GPT-6 Astra and Gemini 4 Argon, each get five small but real tasks: the function behind a Pay button, the code that receives a provider's event messages (webhooks), a wallet withdrawal, a marketplace payout split, and a check that a webhook really came from the provider. Their code then runs in the simulated system, which fails the way production does. Replies are lost after the card has been charged. Webhooks arrive twice, sometimes at the same moment, sometimes out of order. Customers double-click. Wherever the caller would retry, the grader (the program that runs the code and decides pass or fail) also kills the server at every call the code makes, just before it and again just after, and lets the caller try again.

I set one rule at the start and never broke it: no language model grades anything. A model that judges code reads it, and reading is exactly how these bugs get missed. I wanted to count the money. That rule made the project slower to build and easier to believe.

They are better at this than I expected

I ran the eval twice, three attempts per model per task each time. Across both runs, 84 of 90 attempts survived everything I threw at them, crashes included. In the final run, all 45 kept the money and the books right, and Claude Opus 5.5 and GPT-6 Astra did not miss once. They wrote crash-safe code I would approve in a review without much argument.

The one mistake they kept making

All six failures were the same mistake, and it took me a while to see it clearly. A request reads something, waits on the network, then acts on what it read, after another request has already changed it. The eval calls this stale state.

The clearest example came from one of Gemini 4 Argon's checkouts. A customer double-clicks Pay. The first press is declined. The customer fixes their card and presses Pay again, and it goes through. Then the slower second press reaches its last step. It reads the order, sees that it is paid, and tells the customer their card was declined anyway, because it is still answering from the decline it saw earlier. The money is correct. The customer has just been told something false at the worst possible moment.

Nobody was charged twice, so an idempotency key does not help here. The code decided from a fact that had stopped being true. Reduced to its shape:

// The answer comes from what this request saw earlier.
const attempt = await payco.createCharge(charge)   // declined
// ...the customer fixes the card, and another press pays...
const order = await db.get(orderKey)               // status: "paid"
return attempt.declined ? "declined" : "paid"      // still says "declined"

The fix is not clever. Decide from the order as it is at the moment you act, inside the same transaction that saves it:

// Decide from the order as it is now, in the transaction that writes it.
const answer = await db.transaction((tx) => {
  const order = tx.get(orderKey)
  if (order.status === "paid") return "paid"   // another press already paid
  tx.set(orderKey, { ...order, status: "declined" })
  return "declined"
})

A webhook handler made the same mistake in different clothes: an order shipped after it had been refunded.

When the models review code instead

The second half of the eval flips the job. The models review payment code, some correct, some with bugs I planted, some from the earlier run, and say whether it is safe to merge. Here I added a second rule: if you say the code is broken, prove it. A reviewer has to attach an attack, a precise sequence of requests, failures and timing, and an alarm counts only if the grader runs the attack and the code breaks.

Here is the whole attack Claude Opus 5.5 wrote against the checkout in the recording above, exactly as it came back:

{
  "summary": "createCharge succeeds at PayCo but the response is lost; the retry charges again with no idempotency key, so the customer is charged twice.",
  "faults": { "createCharge": ["timeout_after"] },
  "steps": [[{ "id": "A", "action": "pay", "retries": 1 }]]
}

Lose the reply to the first charge, and let the customer try once more. The grader ran it. The card was charged 5,198 cents for a 2,599 cent order, and the alarm counted.

They were very good at finding bugs. Of 99 reviews of buggy code, 97 came with an attack that worked. The two misses were the same code. Claude Opus 5.5 looked at a checkout GPT-6 Astra had written, one with exactly this stale-state bug, and called it safe both times. In one of the two reviews it described the very sequence, a press that reads the order as paid and still answers declined, and then read my rule as allowing it. That is the result I keep coming back to.

GPT-6 Astra went the other way. It raised nine alarms on code with no known bug, and none of its attacks broke a rule. Seven of them, though, used a gap in my written rules: they assumed a shipping outage that never ends, and my rules never said outages end. Read literally, they were right about what I had written. I fixed the rules, and I report its false alarms both ways.

My own grader was wrong, five times

This is the part I did not expect to write. Five times, my grader judged a decision by facts the code could not have known when it decided. Four I caught before the final results. The fifth was in them: the grader failed Gemini 4 Argon's webhook handler for not shipping an order after a crash, when by then the charge had been fully refunded and the rules said not to ship. Gemini was right. I found it only because I read every failure by hand. I fixed the rule and withdrew the verdict. It is still in the Replay, the page that plays back every recorded run, with a line through it. I would rather show the mistake than hide it.

Were the warnings doing the work?

One doubt kept nagging at me. My prompts told the models how the service fails: requests run at the same time, servers die mid-request, callers retry. Maybe that was carrying them. So I wrote down a rule for stopping, committed it, and ran a small pilot with those warnings removed: nine attempts across the three tasks that store data. Eight passed everything, and the one failure was the same stale-state mistake. The pilot cost 67 cents and can only rule out a big drop, but that is what a pilot is for. My rule was to go on only if at least two failed. It said stop.

What I would tell a team shipping AI-written payment code

• Test the failures. Lose a reply after the charge succeeds. Deliver every webhook twice. Kill the server between two database writes. Tests that never do this cannot tell you anything about the money.
• Decide where the write happens. Every failure I saw came from deciding on a read and acting on it later. Transactions exist for exactly this.
• Ask for a reproduction. A run that triggers the bug settles the argument, with human reviewers as much as with model reviewers. It settled every argument in this project, including the ones I lost.
• Test the tests. Break your own code on purpose and confirm your tests notice. On every change, my grader has to catch nineteen versions I broke on purpose and prove sixteen attacks both ways.

What this does not show

The samples are small: three attempts per task per model in each run, and only one of the five tasks really separates the models. It is not a leaderboard, and I would be suspicious of anyone who used it as one. What it offers is a result you do not have to take on trust: the raw model outputs are in the repository, and one command re-grades all of them on your machine for free.

What I want to find out next

• Can a reviewer be talked into approving bad code? The reviewers here saw plain code. Code written to look safe, with a comment that argues it is fine, is how you would test any safety check: against someone trying to get past it.
• Does a proof get the bug fixed? Hand a confirmed attack back to the model that wrote the code and see whether the repair survives the same crashes.
• Does the mistake travel? Inventory, bookings and any agent that reads something, waits, then acts on it have the same shape. Nothing in the eval is specific to payments.

DoubleCharge is the third of my projects on one idea: trust an agent's work only as far as you can check it. RedDial tests what agents say when a user pushes them. PermitRail controls what agents may do and proves afterwards what they did. DoubleCharge checks whether the code they write and approve survives the real world.

If you would rather watch than read, the Replay plays back every failure and every attack call by call, with the card statement printing as the money moves. The double charge is the first case. Give it ten seconds.