Reliability Levers
Production agent reliability is not one problem. An agent run combines model inference, context retrieval, tool calls, external systems, intermediate state, and final verification. A failure can enter through any of those surfaces, then compound as later steps build on earlier state.
That is why the first reliability move is classification. A transient tool failure needs a retry. A missing constraint needs better grounding. A task that keeps drifting needs a different work shape. A high-risk decision needs a checkpoint before execution continues.
This chapter gives you four operator levers for those failure classes:
- Context quality — give the agent the right reality.
- Orchestration — change the shape of the work.
- Independent retries — use more than one attempt when generation is noisy.
- HITL checkpoints — stop bad state from propagating.
The point is not to make agent work deterministic. The point is to know which lever to pull when the output starts drifting.
Classify the Failure Before Changing the Workflow
Production systems do not handle every failure with the same recovery path. Agents need the same discipline. A retry is useful for a timeout; it is harmful for an invalid assumption. More context helps when the agent is missing facts; it can make things worse when the workflow is already overloaded.
| Failure class | What it looks like | Pull this lever |
|---|---|---|
| Missing or noisy context | The agent cannot reliably identify the relevant facts, constraints, APIs, or code paths | Context quality |
| Poor work shape | The run has too many dependent decisions, mixes discovery with execution, or keeps re-deciding scope midstream | Orchestration |
| Noisy generation | The target is clear, but any single attempt may vary in quality or miss a detail | Independent retries |
| Propagation risk | A wrong intermediate decision would contaminate later work or trigger an expensive side effect | HITL checkpoint |
Start with the failure class, then choose the control. Retrying a context problem produces repeated guesses. Adding context to an orchestration problem increases load without shortening the dependency chain. Continuing past a risky decision turns an unresolved assumption into state.
Reliability levers
Four levers shape agent reliability
Why Agent Outputs Drift
An agent step is a probabilistic transformation. It reads a prompt, context, tool results, and prior artifacts, then produces an output that is useful but not lossless.
The output may preserve the important signal. It may also omit a constraint, over-weight a pattern, smooth away uncertainty, or add a plausible detail that was never grounded. That noise is not limited to long sessions. It can enter through one generated sentence, a summary, a plan, a tool interpretation, a code diff, or a handoff between contexts.
The reliability problem starts when generated output becomes input for later work. The next step receives both the useful signal and the noise, then performs another probabilistic transformation on top of it. Repeated reuse turns small distortions into operational drift.
Reliability levers
Generated state compounds risk
Long runs are one way this shows up. Large contexts add distractors, long sessions accumulate generated state, multi-agent workflows pass imperfect summaries across boundaries, and hard tasks contain more high-impact decision points. You do not need the exact probability to use the model. The operational point is enough: dependent transformations need controls that one-shot outputs do not.
Some Mistakes Stick
A mistake becomes sticky when it stops being an output defect and becomes input context for the next step.
If the agent picks the wrong import in step 3, step 4 may call the wrong API. Step 5 may then "fix" the wrong problem because it is reasoning from the bad import. The run is no longer making independent mistakes; it is building on contaminated state.
That is failure stickiness: once one phase goes wrong, the next phase becomes more likely to go wrong too.
Reliability levers
Failures stick until a checkpoint interrupts propagation
This is why phase boundaries matter. A reviewed plan, a fresh-context implementation, or an independent review can stop a bad assumption before it becomes the foundation for the rest of the run.
The four levers control where noise enters, how far it propagates, and when it gets corrected. Context quality reduces noisy input. Orchestration reduces unnecessary dependent transformations. Independent retries handle variance in a bounded step. HITL checkpoints stop noisy state from becoming authoritative.
1. Context Quality: Give the Agent the Right Reality
Reliability levers
Better context raises baseline reliability
Use this lever when the agent is detached from your actual system.
The failure usually looks like this:
- it invents a new cache client instead of using the existing abstraction
- it follows a generic framework pattern instead of your codebase pattern
- it misses a constraint that lives in tests, docs, or nearby code
- it produces an answer that is plausible in general but wrong here
The operator move is to load the facts that constrain this task, not to dump everything into context.
For the rate-limiting task, useful context is specific:
- which middleware pattern the codebase already uses
- how anonymous and authenticated users are identified
- which Redis client or cache abstraction already exists
- what API error shape callers expect
- which routes, roles, or failure modes must be protected
Grounding finds those facts. Context engineering keeps them usable by controlling placement, size, lifetime, and isolation.
What this lever does not fix: a task that is too broad, a vague target, or a bad assumption that has already propagated through the run. Better context raises the starting quality of each step. It does not make long chains safe by itself.
2. Orchestration: Change the Shape of the Work
Reliability levers
Orchestration changes task shape
Use this lever when the agent has enough facts but the work is shaped badly.
The failure usually looks like this:
- the agent starts correctly, then drifts over a large implementation
- one subtask depends on a half-correct result from the previous subtask
- the agent mixes discovery, design, implementation, and cleanup in one run
- the task quietly expands into adjacent refactors
The operator move is to choose the right work unit.
For agents, smaller is not always better. Splitting a task into many serial micro-steps creates more handoff points and more chances for drift. But asking for too much in one shot pushes the model past its capability.
The useful question is:
What is the largest unit the agent can complete reliably without re-deciding the architecture mid-run?
For rate limiting, these shapes behave differently:
- Too broad: "Implement rate limiting."
- Too fragmented: "Find middleware. Stop. Find Redis. Stop. Design limiter. Stop. Add one helper. Stop..."
- Better: "After grounding, implement the limiter in the existing middleware path, reuse the existing cache abstraction, preserve auth behavior, and add the agreed tests."
Orchestration also means removing unnecessary dependencies. If two research tasks are independent, run them in parallel sub-agents instead of forcing one long serial thread. If implementation and review need different judgment, separate them. If a task needs a human decision, stop before code turns that decision into structure.
This is where sub-agents matter. They isolate noisy research or independent candidates, then return a compact synthesis instead of dragging the whole search trail into the main context.
What this lever does not fix: missing facts or an objectively unclear target. Better task shape helps only when each unit has enough grounding and a clear success condition.
3. Independent Retries: Use More Than One Attempt
Reliability levers
Independent retries need selection pressure
Use this lever when one attempt is noisy but the target is clear.
The failure usually looks like this:
- two implementations could satisfy the same spec, but one is cleaner
- the agent sometimes gets the right shape and sometimes misses a detail
- the work is cheap to repeat and easy to judge
- tests, review, or comparison can select the better output
The operator move is to generate more than one candidate and apply selection pressure.
For rate limiting, independent retries might mean:
- ask separate contexts for implementation plans, then pick the least invasive one
- generate two code variants and keep the one that passes tests and matches existing patterns
- retry only the noisy step instead of rerunning the entire workflow
The word independent matters. Asking the same thread to critique itself and then fix its own answer is often not a real retry. The second answer inherits the first answer's framing, blind spots, and contaminated assumptions.
A useful retry has two parts:
- Generate independent candidates. Separate contexts are better than one long self-correction thread.
- Judge with an independent signal. Tests are best when available. Human review is strongest for scope and intent. LLM judges can help when the artifact is small and the criteria are explicit.
What this lever does not fix: weak grounding, a bad task shape, or a missing review boundary. Retrying a poorly specified task just gives you multiple plausible wrong answers.
4. HITL Checkpoints: Stop Bad State from Propagating
Reliability levers
Human checkpoints reduce failure stickiness
Use this lever when a wrong assumption would be expensive downstream.
The failure usually looks like this:
- the plan misses a constraint before implementation begins
- the agent expands scope into cleanup or refactoring you did not request
- a design choice needs product, security, architecture, or migration judgment
- a large diff would be too late to review comfortably
The operator move is to review a small artifact before the next phase begins.
A good checkpoint is not "watch the agent for a while." It is a deliberate gate around a compact surface:
- a short plan before implementation
- a spec before code changes
- a diff summary before merge
- a deployment command before execution
- a fresh-context review before accepting the result
Humans are especially strong at catching missing constraints and scope enlargement. They can see when the agent turned "add rate limiting" into "refactor auth middleware and standardize API errors," even if the diff looks coherent.
The checkpoint only works if the artifact is small enough to read seriously. A checkpoint over a giant plan or unreadable diff becomes theater: the human skims, approves, and the bad assumption continues downstream.
The highest-leverage checkpoints usually sit at phase boundaries:
- after grounding, before planning commits to a direction
- after planning, before implementation
- after implementation, before merge
- before irreversible external actions
A checkpoint is strongest when followed by a fresh phase. The reviewed artifact should become the new starting point, not just another message buried in the same messy thread. Chapter 6's manual handoff pattern exists for exactly this reason.
What this lever does not fix: work that has no clear review surface. If the human cannot judge the artifact quickly, shrink the artifact or split the task.
Optional: The Simple Reliability Model
You can use the four levers without the math. The math is just a compact way to explain why the levers work.
Long Chains Multiply Risk
If a step has some chance of being right, a chain of dependent steps has to be right repeatedly. A mostly reliable step can still produce an unreliable long run.
For example, if each step is correct 95% of the time, a 20-step dependent task succeeds only about 36% of the time:
0.95^20 ≈ 0.358
Do not overfit the number. Real agent work is not a clean sequence of coin flips. The useful lesson is that chain length matters.
Independent Retries Raise Effective Reliability
If one attempt is noisy and retries are meaningfully independent, multiple attempts improve the chance that at least one candidate is good.
For a single step with reliability R, sampled k times:
R_eff = 1 − (1 − R)^k
With R = 0.95 and two independent attempts, the effective step reliability becomes 0.9975. Across 20 steps, that changes the rough chain outcome from about 35.8% to about 95.1%.1
That benefit depends on independence and judgment. If the attempts share the same bad framing, or the judge cannot tell which answer is better, the formula is fiction.
Failure Stickiness Explains Checkpoints
The naive chain model treats each step like an isolated event. Agent runs are often correlated. Once a bad assumption enters the thread, later steps become more likely to build on it.
That is the stickiness parameter S: the chance that a failed step is followed by another failed step.2
| Error stickiness | What it feels like operationally |
|---|---|
| High | One bad assumption contaminates the rest of the run |
| Medium | Some mistakes spread, but reviews or tools catch others |
| Low | Mistakes stay local and the workflow recovers quickly |
Checkpoints, fresh-context reviews, and independent retries reduce stickiness by interrupting propagation.
Choosing the Right Lever
When an agent run fails, diagnose before optimizing.
- If the agent lacked the right facts: improve context quality.
- If the task shape made drift likely: change orchestration.
- If the target was clear but one output was noisy: use independent retries.
- If one bad assumption would poison later work: insert a HITL checkpoint.
Most production workflows combine levers. A spec is both context quality and HITL. Tests are both judgment for retries and verification after execution. Sub-agents are both orchestration and context isolation. Handoffs are both context management and propagation control.
The mistake is treating one familiar mechanism as universal. More context is not always the fix. More decomposition is not always the fix. More retries are not always the fix. More review is not always the fix.
Reliability comes from matching the control to the failure mode.
Key Takeaways
- Diagnose before you fix. Similar-looking agent failures can need different reliability levers.
- Context quality gives the agent the right reality. It fixes missing or noisy facts, not bad task shape.
- Orchestration changes the work shape. It keeps each unit inside the model's capability without creating unnecessary serial dependencies.
- Independent retries help with noisy steps. They only work when attempts are independent and the judge can select well.
- HITL checkpoints stop propagation. They work when the review surface is small enough for a human to judge seriously.
Next: Chapter 8: Spec-Driven Development