── ── AI & agents
Agent Ops Monitoring
An agent that is live without logs, quality metrics or alerts fails quietly. Output quality drifts after a prompt tweak or a vendor model update, spend creeps up, and when something goes wrong nobody can reconstruct what the agent saw, called or said. Most bad outputs are never reported. Customers just leave.
Run Agent Ops Monitoring on a real problem
Bring something you're actually deciding — free, in the browser.
How it works
1. Define logged fields. Per run: trace ID · input · output · tool calls with arguments and results · model and prompt version · tokens/cost · latency · final status. Redact sensitive fields. The OpenTelemetry GenAI semantic conventions give standard names for model, token-usage and tool-call attributes. Gate: a single run cannot be reproduced from its trace → stop and fix logging first. 2. Define quality metrics. Task success rate · human edit/reject rate · complaint/escalation rate · tool-call failure rate. Randomly sample a fixed share of runs each week for scored human review. Gate: a metric cannot be computed from the logs → stop. 3. Set alerts. Quality drift (metric departs from the trailing 4-week baseline) · cost (per-run and daily totals over threshold) · error rate · latency. Each alert has a named recipient. Gate: an alert with no recipient does not count as set. 4. Incident handling and rollback. Classify severity (customer- or money-affecting is highest) → pause or degrade to human handling → roll back to the previous model/prompt version → notify affected parties → run a blameless postmortem. Gate: no one-step rollback to a previous version exists → stop and build version control first. 5. Human-in-the-loop queue. Low-confidence, high-risk and novel inputs go to a human queue with a handling time limit and a backlog cap. 6. Weekly review. Metric trends · sampling findings · alerts and incidents · cost · this week's changes · next week's actions and owners. 7. Stop-rule: if the agent only produces low-risk drafts that can be redone at any time, compress to steps 1 and 6.
Output template: Logged fields · Quality metrics + sample rate · Alerts (condition/threshold/recipient) · Severity levels + rollback steps · Human-queue rules + time limit · Weekly review agenda
→ Method in Action: OpenAI's GPT-4o sycophancy rollback (April–May 2025)
When to use it
- user says 'how do we watch the agent after launch', 'AI output quality is getting worse', 'inference spend suddenly jumped', 'the agent broke something, how do we roll back', 'we need an AI ops weekly review', or an AI agent is already in production handling real business
When not to use it
When the decision is routine and reversible, applying a formal method costs more than it returns.
Worked example
OpenAI's GPT-4o Sycophancy Rollback (April–May 2025)
This case is reconstructed from OpenAI's two public post-mortems: "Sycophancy in GPT-4o: What happened and what we're doing about it" (April 29, 2025) and "Expanding on what we missed with sycophancy" (May 2, 2025). It is the operator's own account of a production quality regression that pre-launch checks missed. Quotes are from those posts.
Install this skill (free, MIT)
npx skills add deciqAI/knowledge-skillsUseful? Star the repo — stars help other builders find it.
Related mental models
AI agents can research every prospect before writing to them, at close to the cost of a mass blast.
An AI shopping agent does not respond to photography, mood or persuasive copy.
AI drives the marginal cost of a draft toward zero, so the bottleneck moves to consistency, accuracy and compliance.
AI does not hit a company evenly.