Trained on a fixed world,
broken by a shifting one.
Researchers at Nanjing University LAMDA asked a direct question: can an agent trained carefully on tool use actually handle a world where the tools, queries, and observations are different? The short answer, confirmed at ICML 2026, is no. Not reliably. And it does not much matter whether training used supervised fine-tuning or reinforcement learning.
First surfaced in Tandemly Briefing — 2026-07-06.
Benchmarks measure
memorization, not adaptation.
Every tool-using agent is trained in some closed environment: a defined set of tools, a corpus of queries, a specific output schema. Then it is deployed into the real world, where none of those things stay fixed.
When you train a tool-using agent, you give it a set of tools with specific names, signatures, and expected outputs. You write or collect training queries. You define what a correct observation looks like when the tool responds. Within that controlled setup, the agent can learn remarkably well. Benchmarks confirm it. You run the agent on held-out examples from the same distribution, and performance looks strong.
The problem is that "held-out from the same distribution" is doing a lot of work there. Real production environments do not hold their distribution constant. APIs update their parameter names. Tool vendors rename endpoints in major releases. Users phrase the same question in dozens of different ways. Tool responses include error codes, partial failures, or unexpected fields. And sooner or later, the agent encounters an entirely new task domain it was not trained for at all.
The question this paper asks is whether standard training approaches, specifically supervised fine-tuning and reinforcement learning, produce agents that can handle this drift. The answer matters not as a benchmark curiosity but as an architectural reality: most production agent deployments assume the trained model will generalize across small environmental variations. The paper tests whether that assumption holds.
Do trained tool-using agents generalize to open environments? Can they handle shifts in how queries are phrased, how tools are named, what observations look like, or what domain they are operating in? And if they cannot, what specifically breaks, and where?
Four shift types.
Four failure tiers.
The team built OpenAgent: a formal framework that independently varies four dimensions of the deployment environment and maps failures to a four-tier hierarchy of agent cognition.
The central contribution of this work is a structured decomposition of how the open world differs from a training distribution. Rather than vague references to "distribution shift," the Nanjing LAMDA team defined four specific shift types that a tool-using agent can encounter after deployment. Each one targets a different layer of how an agent processes a tool-use task.
Each shift type was tested against agents trained via both supervised fine-tuning and reinforcement learning. This matters because both training approaches are used in practice, and their failure modes under distribution shift are not obviously the same. The comparison reveals whether the training method affects which shift types are most damaging.
The team also built a four-tier failure hierarchy to diagnose what, specifically, breaks under each shift. Rather than a single pass/fail score, the hierarchy distinguishes where in the agent's processing chain the breakdown occurs.
To address what they found, the team proposed Perturbation-Augmented Fine-Tuning, or PAFT. The idea: if static training produces fragile agents, the fix is to make the training distribution less static. During fine-tuning, the team deliberately perturbed tool names, query phrasings, and observation schemas, so the agent encountered distribution shifts as part of learning rather than for the first time at deployment.
Both training approaches
break under open shifts.
The assumption that well-trained agents generalize across small environmental variations did not hold. Degradation appeared consistently across training methods and shift types.
Neither training paradigm produced robust open-world agents. Both SFT-trained and RL-trained agents degraded under environmental shifts. The degradation patterns were not identical: the failure mode distribution across the four tiers differed by training approach, suggesting that RL and SFT each produce different fragility profiles rather than equally distributed brittleness.
This is practically relevant. A team choosing RL training to produce an agent that "learns from feedback" should not assume that adaptability extends to environmental shifts in tool naming or observation format. The agent adapts within the training distribution, not across it.
Shifts to the action space, specifically tool name changes, produced some of the most consistent performance drops. Agents trained on specific tool vocabularies failed to identify the correct tool when its label changed, even when the task, available parameters, and expected output were otherwise identical. This implicates the Perception and Interaction tiers: the agent cannot parse its current tool environment accurately when the surface labels do not match training.
This has a direct practical implication. In production, tool APIs change names. If your agent was trained on one API version and the vendor releases a renamed SDK, the agent's tool-call accuracy may drop substantially before you catch it in monitoring.
Perturbation-Augmented Fine-Tuning, which introduces controlled environmental variation during training, reduced fragility across the four shift types. The gains were not uniform: some shift types responded better to perturbation augmentation than others. PAFT is framed as a targeted intervention and the authors are clear that it does not produce full open-world generalization. It is a meaningful step, not a solved problem.
The arXiv abstract for this paper was not accessible during synthesis. This summary is drawn from the ICML 2026 paper's queue metadata, the authors' stated framing, and the four-tier hierarchy and four-shift-type framework the paper formalizes. Specific benchmark numbers and full ablation results are in the paper. The framing here is accurate to the paper's contribution; the precise numerical claims should be verified against the original.
What this means
for building with agents.
Your agent benchmark is a closed circuit if it uses the same tool names, query formats, and observation schemas as training. The gap between your benchmark score and deployment performance is where open-world fragility lives.
Where to go
from here.
If you want to go deeper or apply this directly.