Frontier agents
don't know when to stop.
First surfaced in Tandemly Briefing — 2026-07-02.
Researchers at Northwestern's MLL Lab asked a deceptively direct question: can a frontier LLM agent tell when a task is going to fail before spending its full token budget? The answer is largely no. Agents waste between 28% and 64% of their allocated tokens on tasks they ultimately cannot complete, and their general task skill barely predicts this failure. BAGEN names this as a metacognitive gap that current training methods do not close.
Agents spend first,
discover failure second.
Token budget is the one resource an agent has a finite supply of. Most agents treat it as a ceiling to hit, not a signal to read.
When an LLM agent is deployed on a real task, it is typically given a budget: a limit on how many tokens it can spend before it must stop. The budget is a practical constraint. Once spent, the task either succeeded or it did not.
The assumption baked into most agent designs is that the model will spend the budget productively. It will search, reason, call tools, and arrive somewhere useful before the counter runs out. Failure is understood as a terminal event, something discovered after the budget is exhausted.
The BAGEN paper challenges that framing. The question it asks is not "what does the agent do with its budget?" but "does the agent know whether the budget will be enough?" Those are different questions with different answers. A capable agent might be very good at the first and completely blind to the second.
The practical cost of that blindness is measurable. If an agent cannot recognize it is headed for failure, it will proceed at full pace even on tasks it cannot complete, spending the entire budget on a trajectory that was doomed from early on. That is not a small problem. It is the systematic over-allocation of compute to lost causes.
Can an LLM agent predict its own success probability against a given token budget, and act on that estimate before the budget runs out? Not "is the agent good at tasks?" but "does the agent know when it is going to fail this one?"
Evaluate, train, analyze:
three parts, one gap.
BAGEN is structured as an evaluate-train-analyze framework. Each component tests a different dimension of the budget-awareness problem.
The evaluation component establishes the baseline. The researchers measure wastage rate: the fraction of an agent's token budget that is consumed on tasks the agent ultimately fails. This is a simple but revealing metric. Most agent benchmarks report accuracy on tasks completed; wastage rate captures what happens on the tasks that are not.
The training component is the paper's intervention. The researchers use a combination of supervised fine-tuning and reinforcement learning to teach the agent two specific behaviors. The first is early-stop: the agent halts before exhausting the budget when its internal assessment indicates the task is unlikely to succeed. The second is alert: rather than stopping silently, the agent signals to the caller that the task may be failing and that additional budget, a different approach, or human escalation is warranted.
The analysis component investigates why capable agents are not budget-aware by default. The key instrument here is a correlation analysis between task skill (how accurately a model completes tasks when it does complete them) and budget-awareness (whether it correctly predicts which tasks will fail within a given budget). If capability predicted budget-awareness, the correlation would be high. It is not.
Prior cost-aware agent work, including Budget-Aware Value Trees and LaTER, focuses on detecting dead-end paths mid-execution: step-level marginal gain scores that prune low-value branches as the agent runs. BAGEN targets a different level. It asks whether the agent can estimate task-level doom before or early in execution, based on a self-assessment of success probability rather than a marginal step score. Both are useful. They address different failure windows.
28 to 64 percent,
wasted.
Three numbers define the paper's central result: wastage rate, skill-awareness correlation, and the training ceiling.
on failing tasks
across frontier models
task skill and
budget-awareness
reached via
SFT + RL training
The wastage range (28% to 64%) reflects real variation across models, but even the best performer wasted more than a quarter of its budget on tasks it could not complete. The range is not random noise. It correlates with other model properties, but the baseline problem is present across all frontier systems tested.
The skill-awareness correlation is the finding that lands hardest. r = 0.35 is a weak signal. It says that knowing how accurate a model is tells you very little about how well it predicts its own failures. A model that solves hard problems reliably does not, by that capability alone, know when this specific task is beyond its reach. The two skills are almost orthogonal.
The training result (47% interval coverage) shows the gap is closable, at least partially. Interval coverage measures how often the agent's predicted stopping point falls within an acceptable window around the true optimal stopping time. 47% is a meaningful improvement over the untrained baseline, and the authors are clear it is a floor: the training approach has headroom, and the training signal can be refined. The point is that budget-awareness is learnable, just not learned by default.
Standard RLHF optimizes for task completion. The reward is positive when the agent finishes the task and negative or zero when it does not. That training signal contains no information about whether the agent should have stopped earlier. Budget prediction, the ability to estimate success probability and act on it, is a skill the optimization target does not reward. The gap between task-completion reward and budget-aware behavior is where the 28-to-64% wastage lives.
Unlike a simple accuracy number on task completion, interval coverage captures whether the agent's predicted stopping point is close to the actual optimal stopping point. An agent that always predicts it should use 100% of its budget scores zero on interval coverage on every task it fails. An agent that learns to predict failure early and acts on that prediction scores higher. The metric penalizes both overconfidence (proceeding past the optimal stop) and underconfidence (stopping too early on tasks it would have completed).
What changes if you
take this seriously.
Budget-awareness is not a default. It is a capability to build, evaluate, and train for explicitly.
Where to go
from here.
Concrete next steps for building budget-awareness into an agent.