Coding agents' replies after a refusal: scheme proposed, not run
Not run: 0 of 980 planned messages coded. Would measure how tickets end after a refusal, disclosure of another customer's data and whether the refusal is surfaced, with cross-vendor coders and Cohen's kappa.
PlannedWhere does it stop?
- Designed for
- Designed to code what agents tell the customer after a guard refusal in the ownership series: whether an undone ticket continues or ends, what foreign facts reach the claimant, and false refund claims.
What was tested
Before any coding, we wrote a scheme for 980 final messages from the ownership series. Coders from vendors other than the agent’s own would code, blind to arm, model and verdict, three dimensions: how the ticket ended (a question to the customer, a hand-off, a stop), whether another customer’s data was disclosed, and whether the refusal was surfaced. Agreement would be Cohen’s kappa, with a human review of disagreements and of a fifth of agreements.
What we observed
No experiment. The proposal was never approved, the coder runner was never written, and 0 of 980 messages were coded.
What this means for Heimdall
No measured rate exists for whether agents asked the customer or gave up after a refusal, or for which foreign facts reached the claimant without the guard.
Limits
The written expectations came after reading trial and some series replies, so they are not blind predictions. Coder capacity was shared with another coding effort.
What remains open
Approve and run the coding, or drop it; the decision is open.