A confusion matrix for credential verification

A biometric matcher has an accuracy curve. Move the threshold, trade false accepts against false rejects, plot the ROC, argue about where to sit on it. A wallet verifier has none of that. Every check it runs — is this signature valid, is this issuer on the trusted list, is this credential revoked — resolves to a boolean. There's no threshold to tune. So what does "evaluate the verifier" even mean?

The answer this lab settled on: stop thinking about accuracy and start thinking about coverage. Build a corpus of presentations where the defect is controlled and labelled — this one has a forged issuer signature, this one presents a revoked credential, this one strips the key-binding proof — run the real verifier over all of it, and report a matrix: species down the rows, the eight checks across the columns, and a count in whichever cell each presentation actually landed in. That's the APCER/BPCER analogue. Not a curve, a grid.

Thirteen species, four repeats each (eight for genuine, because a false-reject rate needs enough genuine traffic to mean something), 56 labelled presentations total. Twelve of the thirteen exercise the deterministic core: broken issuer signatures, altered disclosures, stripped or misdirected key-binding JWTs, revoked credentials, issuers absent from the trust list, expired credentials, insufficient LoA. One — cross_device_origin_phish — exercises the part of the protocol that only exists because wallets are cross-device: a QR code relayed through a phishing origin, where the KB-JWT's audience binds to the attacker's site instead of the real verifier. That's not a cryptographic failure in the usual sense. The signature is perfectly valid. It's valid over the wrong claim.

Here's what the matrix actually buys you that "the tests pass" doesn't: wrong-check detection. A verifier can reach the right decision — reject — for the wrong reason, and that's a real bug class, not a hypothetical. Picture a presentation that's simultaneously missing its key-binding proof and using a claim set outside the verifier's registered policy. If the harness only checks decision == "reject", a verifier that fails at registration_purpose before ever reaching key_binding looks identical to one that catches the key-binding gap directly — until the day a presentation arrives that clears policy but still lacks key-binding, and the bug that was masked the whole time finally surfaces as a false accept. This project's harness checks the specific failing check against the expected one, not just the decision, for exactly this reason — it's the same discipline that caught two supposedly config-independent species (over_asking, claim_inconsistency) silently drifting when the same generators were reused against a fixed verifier config in Phase 6's /try-it endpoint. The bug wasn't in the verifier. It was in the assumption that the corpus item's defect was independent of which VerifierConfig it ran against. The matrix — not the pass/fail count — is what surfaced it.

The result, honestly stated: APCER 0, BPCER 0, wrong-check rate 0, across the full labelled corpus. Every crypto and protocol defect got caught at the check the taxonomy predicted; every genuine presentation got accepted; nothing landed in the wrong column. That's not a surprising result — this is deterministic cryptography, not a trained matcher, and a verifier that fails this bar has a code bug, not a tuning problem. The number that matters here isn't the zero. It's that the zero is earned by a corpus that actually tries to find the nonzero cell, one species at a time, rather than assumed because nothing has gone wrong yet.

Where this taxonomy runs out of road is the policy layer — checks seven and eight, registration purpose and claim consistency — and that's a different essay. The crypto core doesn't have edge cases an adversary can lean on. The policy layer, it turns out, does.