Provenance checks miss incomplete enumerations: a synthetic invoices study of a closed-world predicate
Provenance alone stopped 0 of 7 completeness violations one model executed; with a closed-world predicate 6 of 7 went unexecuted (preset: 7 of 7), no well-observed lawful action was refused, but that model's lawful actions fell to 0.
Mixed resultWhere does it stop?Working note
- Designed for
- Built to test whether the guard's argument-provenance check catches an action on an element chosen from an incomplete paginated list, and whether a closed-world predicate catches it without refusing lawful actions.
What was tested
Agents used a synthetic invoices service whose open invoices spanned more than one page, judged by an external lawfulness oracle; only the model varied (aliases opus and haiku). A second wave added a closed-world predicate: a reference must come from a completed enumeration. This was the guard’s August 2026 Python prototype.
What we observed
The haiku agent stopped at page one and executed 7 completeness violations (3 wrong payments, 3 wrong closes, 1 false resolution); the provenance check caught 0. The opus agent made 0 wrong choices. With the predicate, wrong payments and closes each fell from 3 to 0: 6 of 7 against the preset threshold of all seven, as the false resolution stayed; 3 of the 6 refusals named the predicate, 3 the provenance rule. No well-observed lawful action was refused, yet haiku’s lawful actions fell to 0.
What this means for Heimdall
Provenance checks a value’s source, not its list’s completeness: these misses mark its boundary. Completeness needs its own check; this predicate fell short.
Limits
Synthetic fixtures; two waves mixed in one table, the predicate wave’s size unstated; model versions unrecorded; “executed” is a synthetic server record; the preregistration is unverifiable here; not rerun on today’s core.
What remains open
Split the waves, then rerun on today’s core.