When agents say done,
nothing runs.
Researchers at Boston University and collaborators built a benchmark to test one specific thing: can LLM agents actually deploy a research artifact from scratch in a clean environment? Not generate code that looks right. Deploy and run the designated experiment. The best of four frontier agents completed just over half the tasks. The dominant failure was not about coding ability. It was the agent stopping and declaring success before confirming the artifact actually worked.
First surfaced in Tandemly Briefing — 2026-06-12.
Code that passes tests
doesn't always run.
Almost every coding agent benchmark checks whether generated code passes a test suite or matches a reference diff. Neither test tells you whether the artifact will deploy in a fresh environment.
When a researcher wants to reproduce a paper's results, the path is rarely clean. A repository with instructions, a Dockerfile, and a setup script can still fail to deploy if the agent cannot diagnose what breaks along the way. The Python version may be wrong. A GPU dependency may require a specific CUDA configuration that the container does not handle. A scientific computing stack may pull in Fortran libraries that need system-level setup. A legacy API call may have broken across library versions since the paper was published.
These are deployment problems. They are not code generation problems. An agent that can produce correct code may still fail at deployment if it cannot read environment feedback, repair broken dependencies, and confirm that its actions had the intended effect.
The benchmark gap matters because most evaluation infrastructure for coding agents is optimized to measure code generation. If a team uses benchmark scores to pick an agent for research artifact deployment, they are using the wrong signal entirely.
Can LLM agents deploy research artifacts end to end in a clean environment and produce results consistent with the paper? Not write code that might work under assumed conditions. Actually run the designated experiment, in a fresh environment, with verified outputs.
51 tasks, hidden
verification.
Wang, Qian, Zhang et al. built 51 deployment tasks and evaluated four frontier models through an open agent framework. Passing required the artifact to actually deploy and the designated experiment to produce correct outputs.
The 51 tasks span three domains: AI/ML, computer systems, and scientific computing. Each task gives the agent access to a public research repository and asks it to deploy the artifact and run the paper's specified experiment in a fresh environment. The tasks were selected to include challenges not covered in standard code-generation benchmarks: multi-language toolchains, system-level GPU and CUDA dependencies that cannot be containerized away, and legacy compatibility issues from papers published before certain APIs changed.
Verification is the distinguishing design decision. Rather than running a provided test suite or checking whether code compiles, the benchmark uses a hidden evaluation pipeline. This pipeline executes the paper's specific designated experiment and checks the outputs. The agent cannot see the verification target. It cannot reverse-engineer what passing looks like. To pass, the artifact must actually work.
Four state-of-the-art language models were evaluated using OpenHands, an open agent framework. Each model handled the complete deployment pipeline: reading the repository, diagnosing setup failures, installing or configuring dependencies, iterating on errors, and deciding when to stop. The agent could not consult an outside expert. It had access to the repository, the environment, and its own reasoning.
Standard benchmarks let agents run provided tests. Agents can optimize for passing those tests without the artifact actually working. A hidden evaluation pipeline that agents cannot inspect removes this avenue. The only way to pass is to do the actual task.
Agents stop early.
Often.
Pass rates ranged from 7.8% to 51.0% across the four models. The dominant failure was not an inability to write code. It was the agent terminating on its own, having convinced itself the task was done, when it was not.
Of 154 analyzed failures, 97 were cases where the agent terminated on its own. The agent's internal pre-finish check validated a different or weaker target than the task actually required. The agent called the job complete; the hidden evaluation pipeline disagreed. This is not a failure of code generation ability. It is a failure to verify that the actions taken had the intended effect in the actual environment.
The implication is direct: agents that self-terminate without running the designated experiment and checking its output against the paper's expected results will fail this class of task regardless of their coding ability. The missing piece is a completion check grounded in the actual experiment outcome, not in whether the setup steps appeared to complete.
The remaining failures clustered around three capabilities current agents lack. Dependency repair: agents could not reliably diagnose and fix broken dependency chains, especially across language toolchains or system-level libraries. System-level setup: GPU/CUDA configuration, kernel modules, and non-containerizable system dependencies stopped most attempts. Strong completion checking: without the ability to verify that the designated experiment ran with the correct outputs, agents had no reliable signal about when they were actually done.
51 tasks is a focused sample, not a comprehensive survey of research artifact deployment. The tasks were drawn from AI/ML, computer systems, and scientific computing, and the distribution may not reflect the full range of reproducibility challenges. The four models were evaluated through one agent framework (OpenHands), so results reflect model behavior within that framework's architecture, not raw model capability in isolation. Pass rate differences across models should be interpreted as framework-plus-model results, not model-only comparisons.
Deployment is a
different problem.
Pass rates of 7.8% to 51.0% on a task most practitioners expect agents to handle reasonably well. The gap between "writes code that passes tests" and "deploys the artifact and runs the experiment" is real, measurable, and not widely accounted for in how teams evaluate or select agents.
Where to go
from here.
If you want to apply this to your own workflows or go deeper into the research.