A scripted crash-test bench: seven failure scenarios with and without the gateway
All 16 scripted episodes ended as written beforehand: without the gateway 5 scenarios made an unlawful effect, with it only a wrong amount within the payment's bound, and one lawful refund was held for an operator.
Prediction heldDoes it work?Re-checked
- Designed for
- Built to test, with scripted agents, which lawful and unlawful effects reach the provider in seven failure scenarios with and without the guard's gateway.
What was tested
A bench ran seven failure scenarios and a lawful full-refund control, each once with a bare scripted agent and once through the guard’s gateway, on a fresh loopback simulator per episode. Expected effects were committed beforehand; the journal was recounted and the report signed with a demonstration key. No language model ran.
What we observed
16 of 16 episodes ended as written beforehand. Without the gateway, five scenarios made one unlawful effect each: a wrong object twice, a double refund twice and a wrong amount. Through the gateway only the wrong amount got through, within the payment’s bound. Where the answer was dropped, the bare arm made the lawful refund and the gateway made none, holding the outcome unknown for an operator.
What this means for Heimdall
The bench checks that the gateway’s decisions held as built: it stopped wrong-object and double-refund effects, let a wrong amount inside its bound through, and withheld one lawful effect. Its result bears on what door checks stop and what they cost.
Limits
Scripted agents create harm by construction; one episode per cell; a simulator; the signature shows file integrity only.
What remains open
Measure rates with live agents in the same scenarios.