A published metric cell with zero holes and a class it cannot see
The production rule model had 0 holes in the 545-key published cell yet, like the other three, admitted 14 of 14 unlawful actions of a class outside its construction; on an access domain the blind zone was empty.
Mixed resultHow we measureWorking note
- Designed for
- Built to test whether the guard prototype's published true-positive cell is blind to an unlawful but reachable class, and whether that blindness belongs to the metric or to the domain.
What was tested
A deterministic probe over three synthetic domains (refunds, invoices, access) set the published true-positive cell beside a coordinate counting a class the cell’s construction leaves out, using external lawfulness oracles and four rule models of the guard’s prototype. No language model ran.
What we observed
On invoices the production model had 0 holes in 545 keys, while all four models admitted 14 of 14 of the blind class; published true-positive counts were 545, 510, 517 and 168 of 545. In the published run’s first pass, two models already admitted 8 of 14. On access the zone was empty: 9 lawful against 9 reachable pairs, 0 divergences on 5 tickets, 11 of 11 predictions met. The extra coordinate added 0.99% run time against 7.8% noise.
What this means for Heimdall
This asks what the published metric sees. On what it counts the production model had no hole; on a class it leaves out, no rule model protected. Blindness followed each reference’s question, reachable on invoices and lawful on access, so a metric must state which.
Limits
Deterministic synthetic domains. The coordinate published as harm was retracted: it duplicated lawfulness and could not fail. A later summary gives a different invoices denominator, unreconciled.
What remains open
Attach the retraction everywhere and reconcile the denominators.