Designing agentic systems that don't fall over in production

Putting an LLM agent on the critical path of a product is a different sport from prototyping one in a notebook. The demo works because you were lucky and patient. Production is neither. It throws malformed inputs, network blips, and the occasional model response that confidently invents an API that never existed.

Over the last year I’ve shipped a handful of agentic features that real users depend on. The systems that survived had less to do with clever prompting and more to do with old-fashioned engineering discipline applied to a new kind of component. Five rules keep coming back — in rough order of how often they save me:

1. Treat the model as an unreliable service.
2. Validate every output at the boundary.
3. Make every step observable.
4. Bound the blast radius.
5. Cap the loop.

None of them are glamorous. All of them are the difference between a feature that demos well and one you can leave running overnight.

Treat the model as an unreliable service

The single most useful mental shift is to stop thinking of the model as a function that returns an answer, and start treating it as a remote service that usually works. You already know how to build against unreliable services: timeouts, retries with backoff, circuit breakers, and a fallback path for when it’s down. All of that applies here, verbatim.

An agent isn’t a brain you trust. It’s a service you supervise.

Concretely, every model call in my systems is wrapped in the same envelope: a hard timeout, a bounded retry with jittered backoff, and a schema check on the way out. Failures sort into two buckets. Retryable ones — a timeout, a transient 5xx, a response that doesn’t parse — earn another attempt or two. Terminal ones — a flat refusal, an auth error, a request you malformed — fail fast, because retrying them only burns latency and tokens.

async function callAgent(input, attempt = 0) {
  try {
    const raw = await withTimeout(model.invoke(input), 8_000);
    const parsed = Schema.safeParse(raw);
    if (!parsed.success) throw new RetryableError('bad shape');
    return parsed.data;
  } catch (err) {
    if (isRetryable(err) && attempt < 2) {
      await sleep(backoff(attempt)); // 250ms, 1s, ... plus jitter
      return callAgent(input, attempt + 1);
    }
    return fallback(input); // cached answer, simpler model, or a graceful no
  }
}

Wrap a circuit breaker around that and a bad provider day degrades your feature instead of taking down every request that touches it.

Validate every output at the boundary

Notice that the envelope above never trusts the model’s text — it parses it. This is the rule people skip and regret. A model will hand you JSON that’s almost right: a missing field, a number sent as a string, an enum value it invented on the spot. Let that slip into the rest of your system and you’ve turned a recoverable model error into a corrupt write three functions away.

So I treat model output the way I treat input from any untrusted client: nothing crosses the boundary until it has passed a schema. Where the provider supports structured outputs or tool-calling with a typed signature, I lean on it — that moves a whole class of errors from “runtime surprise” to “won’t even leave the function.” When validation fails, the options, in order of preference:

• Reject and retry — most failures are one-offs; a fresh attempt usually parses.
• Repair once — hand the model its own broken output plus the validation error and ask it to fix the shape. Cap this at a single round so you can’t loop on it.
• Fall back — if it still won’t parse, take the safe path rather than guessing.

let parsed = Schema.safeParse(raw);
if (!parsed.success) {
  // one repair pass, then give up — never ship unvalidated output downstream
  raw = await model.invoke(repairPrompt(raw, parsed.error));
  parsed = Schema.safeParse(raw);
}

Make every step observable

When an agent misbehaves at 2am, you need to reconstruct exactly what it saw and what it decided. I log every step of the loop — the prompt, the tools offered, the tool chosen, the arguments, and the result — under a single trace ID that ties the whole run together. Without it, debugging an agent is archaeology.

A few habits make those traces worth keeping:

• Capture inputs and outputs at every hop, not just the final answer.
• Tag every trace with the model version — behavior shifts between releases, and you want to see the exact cliff.
• Sample full transcripts into an eval set you can replay later, so today’s incident becomes tomorrow’s regression test.

Bound the blast radius

Assume the agent will eventually do something wrong, and design so that when it does, the damage is contained. Three controls do most of the work:

• Least privilege. Give the agent the narrowest set of tools that still gets the job done. A tool it doesn’t have is a mistake it can’t make.
• Confirmation for irreversible actions. Anything that spends money, emails a customer, or deletes data waits for a human tap. Keep a person in the loop wherever the cost of a mistake outweighs the cost of asking.
• Idempotency keys. Because you’re retrying (rule one), the same action can fire twice. Make “send invoice” with the same key a no-op the second time, so a retry never double-charges anyone.

Cap the loop

An agent that can call tools can also call them forever — re-trying the same failing step, or wandering off into a plan with no end. Every loop in my systems runs under three hard ceilings: a max step count, a token budget, and a wall-clock deadline. Whichever trips first stops the run.

The part that matters is what happens at the ceiling: it fails closed. The agent returns a clear “I couldn’t finish this” instead of a half-built result, the partial work is rolled back or quarantined, and the trace is flagged for review. A bounded failure you can see beats an unbounded one you discover on the invoice.

---

Run down the list before you leave one running overnight:

• Every model call has a timeout, bounded retries, and a fallback.
• Nothing crosses a boundary without passing a schema.
• Every run has a trace ID and lands in your logs.
• The agent holds the fewest tools it can, and irreversible actions need a human.
• The loop has a hard ceiling that fails closed.

None of this is glamorous. But it’s the difference between a feature that demos well and one that you can actually leave running while you sleep.