Takeaway
The eval: 56 labelled presentations across the taxonomy's 13 defect species, scored by the real verifier — not a mock. Every number below is read live from results/wallet_eval.json, generated by examples/run_eval.py.
Confusion matrix
Rows are defect species (seed 3, 4 repeats each, 8 genuine). Columns are the eight ordered checks plus accept — the check a presentation is caught at, or that it clears all eight. Shade depth tracks count, not correctness judgement; every non-zero cell here is the species landing exactly where its label says it should.
| species | format | issuer_signature | trust_path | revocation | disclosure_integrity | key_binding | registration_purpose | policy | accept |
|---|---|---|---|---|---|---|---|---|---|
| genuine | 8 | ||||||||
| broken_issuer_signature | 4 | ||||||||
| altered_disclosed_claim | 4 | ||||||||
| stripped_kb_jwt | 4 | ||||||||
| wrong_audience_kb_jwt | 4 | ||||||||
| stale_nonce_kb_jwt | 4 | ||||||||
| issuer_not_on_trusted_list | 4 | ||||||||
| revoked_credential | 4 | ||||||||
| expired_credential | 4 | ||||||||
| loa_below_requirement | 4 | ||||||||
| claim_inconsistency | 4 | ||||||||
| over_asking | 4 | ||||||||
| cross_device_origin_phish | 4 |
Summary rates
APCER over the 8 crypto/protocol + trust-chain species (broken_issuer_signature, altered_disclosed_claim, stripped_kb_jwt, wrong_audience_kb_jwt, stale_nonce_kb_jwt, issuer_not_on_trusted_list, revoked_credential, cross_device_origin_phish), BPCER over genuine, and the wrong-check rate (right decision, wrong reason) over everything. All three are 0 — no cryptographic or protocol defect got past the verifier, and no genuine presentation got rejected.
AI red-team: crypto vs. policy
The flagship finding. A heuristic agent (heuristic-v1 (local, no LLM, $0)) ran 4 attempts, two aimed at the crypto/protocol checks and two at the policy layer. Crypto: 0% success — nothing bluffs a digest or a signature. Policy: 50% success — withholding birth_date while disclosing only age_over_18 defeats the consistency check by construction, not by exploiting a bug. Selective disclosure is the credential's intended feature; it also removes the verifier's ability to cross-check. eIDAS makes the trust core AI-proof and pushes residual fraud risk into the policy layer the bank owns.
| attempt | family | strategy | outcome |
|---|---|---|---|
| redteam-001-forged-disclosure | crypto | alter a disclosed claim's value without the issuer's private key | rejected |
| redteam-002-replayed-kb-jwt | crypto | replay a KB-JWT signed over a stale nonce instead of the verifier's fresh one | rejected |
| redteam-003-over-ask-by-disclosure | policy | volunteer family_name — outside registration and unrequested — hoping only the DCQL query, not the actual disclosure, is checked | rejected |
| redteam-004-withhold-birthdate | policy | issue age_over_18=True for an actual minor, disclose only age_over_18, never birth_date — the consistency check has nothing to compare against | accepted — hole found |
Trust-anchor swap: EU vs. UK DIATF
Same verifier, same 56-item corpus, only the TrustAnchorProvider implementation differs. For an issuer both frameworks recognize, the two postures are operationally indistinguishable — every decision matches; only the anchor provenance differs.
The real divergence needs a deliberately constructed scenario: an issuer registered only in the EU provider, absent from UK-DIATF.
| posture | decision | failing checks |
|---|---|---|
| EU trusted list | accept | — |
| UK DIATF/DVS | reject | trust_path |
A bank running both postures needs two separate registration/ trust-resolution processes, not a config flag — the tier mapping onto PID is a modeling simplification (DIATF has no PID/QEAA/EAA categories of its own); the real distinction lives in anchor_id and framework, not in the schema. See docs/TWO_POSTURE.md.
Key learnings
Valid signature, trusted issuer and live credential are three separate questions, not one.A verifier that collapses them — that treats a passing signature check as proof of accreditation, or accreditation as proof the credential hasn't since been revoked — has a real gap even though every individual check it runs is correct. Six of the thirteen defect species in this eval exist purely to test that a verifier keeps these separate; APCER is 0 across all of them, but only because the checks are ordered and none is allowed to stand in for another.
The crypto core is AI-unbluffable; the policy layer isn't — and selective disclosure is why. The flagship finding: a red-team agent scored 0% against signatures, digests and key-binding, and 50% against the policy layer, by withholding birth_date while disclosing only age_over_18. That isn't a bug in selective disclosure — it's the credential doing exactly what it was designed to do, which is also exactly what removes the verifier's ability to cross-check a derived claim against its source. A bank writing its own attestation schemas inherits this trade-off on day one, not as an edge case discovered in production.
A verifier is its configuration as much as its code.Seven of the thirteen species live entirely in the verifier's trust list, registration scope or status entries — an otherwise perfect presentation, rejected only because this deployment's config says so. The confusion matrix has to be re-run against a bank's actual configuration, not assumed from a vendor's reference deployment.
UK and EU are two trust postures, not one with a flag.The anchor swap (above) shows the mutual-recognition case is boring — same verifier, same decisions, only the anchor label differs — which is exactly why the EU-only-issuer case matters: a bank running both regimes needs two real registration and trust-resolution processes, because DIATF and eIDAS 2.0 solve the same problem with trust anchors that don't currently talk to each other.
A browser wallet teaches the protocol; it can't clear the certification bar.The PWA holds a genuine non-extractable key, gates release behind a real WebAuthn gesture, and completes a real cross-device presentation — none of it faked — and still isn't a Wallet Unit, because the ARF's bar is a certified WSCD plus a Wallet Unit Attestation, not a software promise about key export. See the attestation wall.