A 10-seed label-permutation control is unreliable; an exact replacement
With 1,000 seeds, 5 of 12 blocks were only conditionally certified (1 or 2 expected); an exact criterion changed 0 of 12 verdicts but took 13.897 s against 0.019 s, refuting the prediction that cost would not rise.
Prediction failedHow we measureWorking note
- Designed for
- Built to test whether the label-permutation arm that certified each published block of the guard prototype's probe is a valid and stable control, and what should replace its random draw.
What was tested
We reran the 12 frozen probe commands with 1,000 seeds for the label-permutation arm instead of 10, then replaced the random draw by an exact type-I error criterion at a preset significance level. Synthetic domains, four rule models, no language model.
What we observed
5 of 12 blocks were only conditionally certified, against 1 or 2 expected. Margin did not track fragility: the tightest arm (0.0545 points) fired 0 of 1,000 times, the widest (3.8471) 30 of 1,000. The first 10 seeds gave 6 shots against 2.09 expected. 4 of 16 arms were duplicates. The exact criterion changed 0 of 12 verdicts, dropped “conditionally” from 5, and refused an arm the old draw passed; it took 13.897 s against 0.019 s, refuting the prediction that cost would not rise.
What this means for Heimdall
This asks whether the control certifying the probe’s published blocks was sound. Its random draw was unstable, so a random control needs its seed sensitivity measured first. The exact replacement changed no block’s verdict.
Limits
An internal instrument whose shuffle arm is not in the published cells; a residual reservoir draw still moves the printed number; base code outside this repository.
What remains open
None planned; the exact criterion replaced the draw.