The benchmark score is also
a harness score.
First surfaced in Tandemly Briefing — 2026-06-13.
Researchers at Beijing University of Posts and Telecommunications, Shanghai AI Lab, and collaborating institutions ran 15 LLMs across 7 widely-used agent benchmarks. They used a single standardized harness for every model and every benchmark. What they found: published scores contain two variables mixed together. The model is one of them. The scaffold is the other.
Every benchmark ships
its own scaffold.
That means every published score is actually a score of model plus scaffold. Researchers and teams making decisions from those scores are reading a compound number as if it were a simple one.
AI agent benchmarks work like this: a team publishes a dataset of tasks, writes the infrastructure to run models against those tasks, and reports how well various models perform. The benchmark covers things like web navigation, code execution, multi-step reasoning, or tool use. It becomes a standard comparison point.
The problem is that the infrastructure, the harness, is not neutral. Each benchmark team makes choices about how the agent is prompted, how tool calls are formatted, whether the environment is reset between attempts, how failures are detected, and how partial completions are scored. These choices are engineering decisions, not science ones. And they interact with model behavior in ways the benchmark team did not necessarily intend or test.
The result: when model A outperforms model B on benchmark X, that score could mean model A is genuinely better at the task benchmark X was designed to measure. Or it could mean model A is better-matched to the particular scaffold benchmark X uses. Without a controlled comparison, you cannot tell which.
For production teams choosing between models or evaluating their own agent stack, this is more than a methodological inconvenience. It means that two benchmarks showing different rankings for the same models could both be correct, and still tell you very little about which model performs better in your specific setup.
If you run many models through many benchmarks but control the harness across all of them, how much do the rankings change? And can you separate the model's contribution from the harness's contribution cleanly enough to be useful?
One harness.
Seven benchmarks.
Fifteen models.
The team built a unified evaluation framework that converts seven different agent benchmarks into a single instruction-tool-environment format, then runs every model through a fixed ReAct loop. The benchmark's original scoring criteria are preserved; the scaffold is not.
ReAct (Reasoning plus Acting) is a standard loop structure for agent evaluation: the model receives a task, reasons about what to do, takes an action via a tool, observes the result, and reasons again. Different benchmarks implement this loop differently. The unified framework standardizes the loop while leaving each benchmark's underlying task set and success criteria intact.
Seven benchmarks were adapted to this format, covering 24 domains. The selection spans single-agent tasks, multi-agent coordination, and safety-critical scenarios, so the framework is not optimized for any one class of problem. Fifteen models were evaluated across these benchmarks using the unified harness.
The framework also includes an offline snapshot mode. In ordinary evaluation, some benchmarks involve live environments that change between runs: web pages that update, APIs that go down, services that return different results. This volatility contaminates comparisons because it introduces a source of variance that has nothing to do with the model. The offline mode replaces live environments with curated, stable snapshots, letting the researchers isolate environment effects from model effects.
The framework separates two things prior benchmarks merged: the task specification (what success looks like) from the infrastructure (how the model interacts with the environment). It keeps the former intact and replaces the latter. This lets the researchers attribute performance differences to a single variable at a time.
Unified resource metrics (token usage, latency, cost) and failure attribution were added across all benchmarks. Previously, comparing resource consumption across benchmarks required reconciling different reporting conventions. The framework makes these comparisons direct.
Scaffold choice moves
outcomes both ways.
The headline finding is not that harnesses matter more than models. It is that they are not separable in published evaluations, and the gap is wide enough to reorder leaderboards when you control for one and vary the other.
When the same model is evaluated through different harnesses on the same task set, performance can shift up or down. This is not a small correction. It is large enough to change which model ranks first. A model that appears dominant on a benchmark using that benchmark's native scaffold may rank lower under a different harness, and vice versa.
This is not a claim that one harness is better than another. It is a claim that scaffold choice is a variable, and published evaluations do not treat it as one.
The offline snapshot mode isolates what happens when the same model runs the same benchmark in a stable environment versus a volatile one. Environmental drift, API changes, and service variation contribute to score variance that is unrelated to model capability. The framework makes this variance measurable for the first time across a common set of benchmarks.
For teams running internal evaluations against live environments, this finding suggests their results carry a noise floor they may not have accounted for.
By standardizing resource reporting and failure logging across seven benchmarks and fifteen models, the framework produces the first cross-benchmark failure attribution analysis at this scale. Teams can identify whether a model's failures concentrate in specific task types, tool use patterns, or multi-step coordination scenarios. This is difficult to do when each benchmark produces different logs in different formats.
The framework standardizes the harness but cannot eliminate all evaluation confounds. The benchmark tasks themselves were designed by different teams with different goals. A unified harness reduces one source of noise; it does not eliminate differences in task distribution, difficulty calibration, or annotation quality between benchmarks. The paper is transparent that the goal is disentanglement, not elimination of all variance.
What this means
for choosing and
evaluating models.
The core implication is not complicated: before trusting a published leaderboard to inform a model selection decision, ask what harness produced it. Then run the candidates through the harness you will actually use.
Where to go
from here.
Next steps if this work is relevant to how you evaluate or select models.