── ── AI & agents
AI Output Verification Trust Tiers
Teams usually make one of two mistakes with AI output. They trust all of it — and fabricated figures, citations or terms reach a client — or they review all of it, and the AI saves no time. Both are wrong, and the second quietly becomes the first: reviewers who check everything start approving without checking.
Run AI Output Verification Trust Tiers on a real problem
Bring something you're actually deciding — free, in the browser.
How it works
1. List output types. By use: client emails, quotes, data summaries, code, citations, payment instructions, and so on. Gate: a vague type ("the AI's answers") → split it → stop. 2. Score two dimensions. Error cost (low / medium / high: money, compliance, client relationship, irreversibility) · verifiability (can it be checked quickly against a rule, a source document, or a calculation?). 3. Assign the tier. - T1 Automatic: low cost + easy to verify → rule-checked, then released; periodic spot-checks - T2 Sampled: medium cost, or verification needs a human → a fixed share sampled for human review - T3 Mandatory review: high cost → every item reviewed by a human before release - T4 No automation: high cost + irreversible (payments, signing, external commitments) → AI drafts only; a human executes Gate: an irreversible action placed in T1 or T2 → stop and re-tier. 4. Attach verification methods. Source check (every citation must open to the original) · rule / format validation · recalculation · cross-model or regenerate-and-compare · human spot-check. Gate: T3 or T4 with no evidence requirement → stop. 5. Set the evidence requirement. Facts, numbers and citations carry a source link or source-document location. No source = unverified. 6. Set escalation. Low confidence, amount over threshold, new output type, client complaint, or anyone disputing the output → move up a tier or route to a human; name the reviewer. 7. Re-tier with data. Move tiers up or down on spot-check error rates; moving down requires several consecutive periods meeting the bar. 8. Stop-rule: every output stays internal and can be redone at zero cost → collapse to T1 plus spot-checks.
Output template: output type · error cost · verifiability · tier · verification method · evidence requirement · escalation trigger + reviewer · spot-check rate
→ Method in Action: Mata v. Avianca — the fabricated citations (2023)
When to use it
- user says 'can we use the AI's output directly?', 'what needs human review and what doesn't?', 'how do we verify AI output?', 'the AI made up data again', or needs review rules for AI outputs, or is deciding where an agent may act automatically and where a human must sign off
When not to use it
When the decision is routine and reversible, applying a formal method costs more than it returns.
Worked example
Mata v. Avianca — The Fabricated Citations (2023)
A worked example of a high-cost, easily verifiable AI output released with no verification at all — documented in Judge P. Kevin Castel's Opinion and Order on Sanctions, U.S. District Court for the Southern District of New York, June 22, 2023 (22-cv-1461 (PKC), ECF 54). Every fact below is from the court's findings. The case is useful because nearly every fake move in this skill's rationalization table appears in the record.
Install this skill (free, MIT)
npx skills add deciqAI/knowledge-skillsUseful? Star the repo — stars help other builders find it.
Related mental models
Most AI pilots do not fail on the model.
Small teams rarely get hurt by exotic AI risk.
AI business cases are usually inflated the same three ways: they count only "hours saved," they omit the costs that arrive after the subscription (inference, integration…
The core mechanism: build only the layer that differentiates you and depends on data only you hold; buy or partner for everything else — and put…