Not every agent in your pipeline should be an LLM call

I keep seeing pipelines where every step goes through the model. Validation. Formatting. Retries. The final write. All of it routed through an LLM call, because the whole thing got labelled an agent and agents call models.

That is expensive, slow, and non-deterministic in exactly the places that should never vary.

The wrong question

The design decision is not agentic against not agentic. That framing puts the whole pipeline on one side of a line, and pipelines do not work that way.

The decision happens sub-agent by sub-agent: which step needs judgment, and which step just needs correct, repeatable execution.

The test I use

One question routes every step.

Could you write a test case for the correct output? A spec routes to code. A judgment call routes to the model.

If you could write a test case for the correct output, it is a spec, and a spec belongs in a function. Deterministic, testable, cents per run.

If "correct" is a judgment call rather than something you could specify in advance, that is what the model is for. Dollars per run, and worth it.

The uncomfortable part is how many steps fail that test in the direction nobody expects. Validation is a spec. Formatting is a spec. Retry logic is a spec. Those are the three I most often find sitting inside a prompt, and all three have a correct answer you could have written down before the run started.

Thin harness against fat harness

There is a useful pair of terms for the two shapes this produces.

A thin harness has the model design the pipeline once. Code runs it every time after that. The judgment happened at design time and got captured in something repeatable.

A fat harness has the model replan the whole task on every run. Every execution pays for the planning again.

The model appears once on the top row and on every run below it. That difference is the cost, the latency, and the variance.

Both are correct choices, for different problems.

Coding tools like Claude Code and Cursor go fat harness deliberately. The task genuinely is different every time, and novelty is the point. Replanning is the product.

Most shipping SaaS AI features should be thin. The task repeats, the shape is known, and a wrong answer compounds across every run rather than being caught by the person watching.

What the reliability data says

The case for keeping the model out of repeatable steps gets stronger the longer your pipeline runs.

A 2026 reliability study of long-horizon agents measured aggregate pass@1 across ten models and four task-duration buckets: 76.3% on short tasks, falling to 52.1% on very long ones. Individual results are harsher. On TensorBench, several coding agents dropped from roughly 48% to 12%, and from 40% to 14%, once task descriptions got long.

Source

Aggregate figures from Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents, which defines short as under five minutes of human completion time and very long as over two hours. Per-model coding figures from TensorBench. Both measure autonomous performance without a deterministic harness, which is the point.

The longer and more open-ended you let a run become, the less you can trust the model alone to hold it together. Every step you move out of the model shortens the horizon it has to survive.

The consequence

A flaky agent usually does not need a bigger model. It needs less work handed to the model.

Let code handle the part that is just repeatable steps. Save the model for the part that actually needs judgment. More model calls is not more intelligence, and past a certain point it is just more latency and a bigger bill with more surface area for variance.

Look at the last agent you shipped. Find the step that was a pure LLM call where a deterministic function would have done the same job, every time, for a fraction of the cost. There is almost always one.