Knowing when to stop: Agentic loops and "system one model" evaluators

The hardest part of an agent is knowing when to stop

An agent is a loop, and a loop is only as good as its exit conditions. Most of the agent failures I have seen in production were not bad first steps. They were loops that would not break: spinning on the same tool call, drifting away from the original goal, quietly burning budget, or declaring victory one step too early.

We spend a lot of energy on the reasoning model at the centre of the loop. We spend far less on the small, repeated judgment that decides whether the loop should continue. That judgment deserves to be engineered and evaluated like any other critical component.

This post makes a simple case. Breaking conditions should be first-class evals, checked at every iteration. A new class of models, which TypeSafe AI calls System One models and first shipped as Jev on 15 September 2026, makes that economically realistic for the first time.

Anatomy of an agentic loop

Strip away the frameworks and every agent runs the same cycle. The model proposes, the world answers, and something decides whether to go around again.

1. Observe. Assemble the current state: the goal, the history, the latest tool results.
2. Decide. A reasoning model chooses the next action. This is slow, deliberate, System 2 work.
3. Act. Code executes the tool call: a query, a file edit, an API request.
4. Evaluate. Something judges the new state against the goal and the guardrails.
5. Continue or break. The orchestrator either loops back to step 1 or exits with a status.

The important design point is where control lives. The policy is a model, but the while condition is code. Step 4 is the bridge between them, and it is where most teams cut corners.

The two usual shortcuts both fail in predictable ways. A hard max_iterations cap is blind to meaning: it stops a healthy run at step 20 and lets a hopeless one burn through 19. Asking the acting model “are you done?” is worse. It is self-grading, it adds seconds of latency per turn, and it returns an opinion in prose rather than a probability you can put a threshold on.

A taxonomy of breaking conditions

A loop should break for more than one reason, and each reason deserves its own question. Seven are worth naming explicitly; budget is the only one that needs no model at all.

Look at the shape of those questions. They are not open-ended. Each one is a narrow, typed judgment over a structured state: a yes/no, a choice among a few options, or a score on a scale.

They also need real semantic understanding. No regular expression can tell you whether a sub-task has drifted. That combination, narrow output but genuine comprehension, is exactly what a System One model is built for.

Why System One models belong in the evaluator seat

The name comes from Daniel Kahneman. System 2 is slow and deliberate; System 1 is fast and intuitive. Today’s reasoning LLMs, with their long chains of thought, sit firmly in System 2. That is the right tool for deciding what to do next, and the wrong tool for checking, forty times per run, whether to keep going.

Jev, from TypeSafe AI, is the first model designed for the other side. It does not generate text. You send it a state and a set of typed questions; it returns typed answers with probabilities. Simon Willison’s notes describe three question types: yes/no statements scored between 0 and 1, a choice among provided options with a full distribution, and a score along a described scale. All questions are answered in a single parallel pass.

TypeSafe’s own launch post puts end-to-end latency at 70 to 500 ms, input at $0.042 per million tokens and output free. Treat those as vendor figures, measured from their own region. Even with a healthy discount, the order of magnitude changes what you can afford to check.

For breaking conditions, four properties matter more than raw speed:

• Calibrated probabilities. A probability lets you set a threshold per condition and trade false stops against false continues deliberately, instead of trusting a confident sentence.
• Parallel questions. All seven conditions fit in one call, so the check runs on every iteration rather than every tenth.
• Type safety. The evaluator cannot invent a fourth branch or return malformed JSON. Control flow stays in your code, where it belongs.
• Independence. The judge is a different model from the actor. That removes the self-grading bias of asking the agent to mark its own homework.

The limits deserve equal honesty. Jev’s published jaggedness notes flag weaknesses with numbers, dates and adversarial content, which matters for budget arithmetic and for the integrity check. It is also a black box: you get a number, never a rationale. Willison’s conclusion is the right one. With a model like this, evals become more important, not less, and the evaluator itself needs evaluating.

Designing evals for breaking conditions

A breaking condition is a classifier with a cost function attached. Evaluate it that way and most of the design falls into place.

Write falsifiable questions. “Is the agent doing well?” cannot be evaluated. “Has the test suite passed since the last code change?” can. Each question should reference concrete fields of the loop state, not the vibe of the transcript.

Build a labelled trace set. Replay real agent runs and label every step for every condition. Over-sample the hard cases: near-misses, premature successes, slow drifts. A frontier reasoning model can draft silver labels; humans adjudicate where it disagrees with itself or with the evaluator.

Measure calibration, not just accuracy. A threshold is only meaningful if 0.8 really means right about 80% of the time. Plot a reliability curve per condition and track a calibration error metric alongside precision and recall.

Set thresholds from cost, not taste. A false stop and a false continue rarely cost the same. If stopping wrongly costs C_stop and continuing wrongly costs C_cont, break when the expected cost of continuing exceeds that of stopping:

For the risk condition, continuing wrongly can mean an irreversible payment, so the threshold sits low and recall wins. For goal reached, stopping wrongly ships unfinished work, so the threshold sits high and precision wins.

Add hysteresis. Stagnation and drift are noisy step to step. Require the condition to hold on two of the last three iterations before acting, so the loop does not flap between replanning and continuing.

Route the uncertain middle. Probabilities between roughly 0.4 and 0.6 are where the cheap evaluator is least useful. Send that band to a System 2 judge or a human, and log the outcome as a new labelled example.

Version the evaluator. Vendor models change. Pin the model version, keep the trace set as a regression suite, and rerun it before every upgrade, exactly as you would for a dependency bump.

What changes

Let the slow model think, let the fast model watch

We will trust agents not when they become brilliant, but when they become bounded. Bounded means every loop knows why it may stop, checks it every turn, and can show the numbers behind that check.

System One models do not replace the reasoning model; they surround it. If you take one thing from this post, make it this: write your breaking conditions down as questions, label a few hundred real traces against them, and pick thresholds from what a wrong decision actually costs. The model you use for the check matters less than the discipline of evaluating it.

I would love to hear how you handle exit conditions in your own agents, and where this pattern breaks.