Agents in production: the boring parts nobody demos
The demo is genuinely easy now. An afternoon gets you an agent that reads your inbox, drafts sensible replies, and looks like the future. Everyone who has shipped one into production knows what happens next: that afternoon was about fifteen percent of the work, and none of the remaining eighty-five is interesting enough to demo.
Show the data
| Phase | Share of effort |
|---|---|
| Get it working at all | 15% |
| Failure handling + retries | 25% |
| Evaluation + regression set | 25% |
| Cost + rate limits | 15% |
| Observability + escalation | 20% |
Five problems that are not about prompting
- 01Non-determinism meets side effects.
The same input can produce a different action. That is fine for a draft and unacceptable for a send, a refund, or a write to a customer record. Separate deciding from acting: the model proposes, deterministic code validates and executes, and every action is idempotent so a retried run cannot email someone twice.
- 02Cost has no natural ceiling.
A retry loop with a large context is a way to spend money at machine speed. You need a per-run token budget, a hard cap per period, and an alert when spend per unit of work moves — because a prompt change can double it silently.
- 03Rate limits and outages are normal operation.
Backoff, a bounded retry count, and a defined answer for what happens when you exhaust it: queue, degrade, or fail loudly. “It throws” is not an answer when the thing runs unattended at 3am.
- 04Quality drifts without a regression set.
You improved the prompt — compared with what? Twenty to fifty real cases with known-good outcomes, run before every change. It is tedious and it is the only thing standing between you and fixing one case while breaking four.
- 05Confidently wrong is the default failure.
The model will not tell you it is unsure; it will produce something plausible. So define what unsure looks like from the outside — low-confidence signals, out-of-distribution inputs, actions above a threshold — and route those to a human before they act.
Pick a job with a cheap failure
The gap between demo and production narrows enormously if the first agent's mistakes are recoverable. Two versions of the same idea:
| Drafts replies | Sends replies | |
|---|---|---|
| What it does on success | Saves six hours a week | Saves seven hours a week |
| What a mistake costs | A bad draft nobody sends | A wrong email to a customer |
| Who notices a mistake | The person reviewing | The customer |
| Can you roll it back? | Yes, trivially | No |
| Needs a human in the loop? | Yes, by design | Only after the damage |
The left column delivers most of the value and almost none of the risk, and it produces something the right column needs anyway: weeks of human judgement on real outputs, which is your regression set and your confidence threshold, collected for free.
Log the decision, not just the output
When someone asks why the agent did that, you need an answer in minutes. That means recording, per run:
- The input, exactly as received — not a summary of it.
- The prompt and model version actually used, so a behavior change can be tied to a deploy.
- What was proposed, what the validator allowed or rejected, and the reason.
- Tokens and cost, per run, so the spend chart is a chart and not a monthly surprise.
- The outcome, including whether a human overrode it — overrides are your highest-value training data.
This is ordinary software engineering
Retries, idempotency, budgets, regression tests, audit logs, human escalation. None of that is new, and none of it is AI-specific — it is what shipping anything that runs unattended and touches money has always required. The model is a component with unusual failure characteristics, not a different discipline.
Which is good news, because it means the hard part is something that has known answers. The reason agent projects stall at the demo is not that the remaining work is unsolved. It is that it is unglamorous, and nobody budgeted for it.
The demo proves it can work. Production is the argument that it will keep working on a Tuesday when the API is slow and the input is weird.
One week, one agreed deliverable: the failure handling, the cost ceiling, the regression set, and the escalation path. In your repo by Friday.
See how the free week works →