The guard did not save model spend: a cost reading of 317 saved sessions
In the one resolvable contrast (n = 24), sessions with the guard on cost 1.14 times those with it off (1.03 to 1.26), on different builds; ablation arms showed no resolvable difference, and no saving was found.
Prediction failedWhere does it stop?Working note
- Designed for
- Built to test what the guard costs or saves in model spend, and whether it pays for itself in tokens.
What was tested
A cost reader took the agent CLI’s own result records from 317 saved sessions, using each session’s maximum because the field is cumulative, and compared guard-on and guard-off arms with 95% intervals. The true contrast paired a guard-on and a guard-off wave with the same prompt and model.
What we observed
In that contrast (n = 24), sessions with the guard cost 1.14 times as much (1.03 to 1.26). Ablation arms removing only some rules gave 1.03 (0.96 to 1.11, n = 49), and 1.05 (0.92 to 1.20) on 25 pairs after removing rollbacks. Replacing an LLM judge with the guard’s verdict fell short of the preset threshold (0.42 times). Naively summing cumulative records overstated spend 1.41 times.
What this means for Heimdall
This asks what the guard costs in model spend, not whether its checks stop harm. In this corpus it was an added cost, not a saving, so its case cannot rest on saved tokens.
Limits
The contrast waves ran different builds, so overhead and build effects cannot be separated; the repeat-noise floor (median 1.33 times, maximum 3.12) shares no wave with the contrast; the corpus is outside the repository.
What remains open
Publish or hash the corpus, or mark it not reproducible.