DeployBench: Benchmarking LLM Agents for Research Artifact Deployment

LLM agents have made rapid progress on software engineering and ML research tasks, but these advances often assume access to a working runnable environment. For research artifacts released alongside published papers, setting up such an environment from a fresh machine remains a major bottleneck. Existing environment-setup benchmarks do not cover the full scope of research-artifact deployment, which involves multi-language toolchains, system-level dependencies beyond containers (e.g., GPU/CUDA and kernel configurations), and legacy artifact compatibility.

We introduce DeployBench, a multi-domain benchmark of 51 research-artifact deployment tasks drawn from top-tier venues published between 2008 and 2025, spanning AI/ML (25), computer systems (19), and scientific computing (7). The tasks cover 11 programming-language ecosystems and include 22 GPU-dependent workloads, 5 tasks that compile and boot custom Linux kernels inside a QEMU virtual machine, and 10 legacy artifacts (2011–2018) that require compatibility repair against modern toolchains. By setup difficulty, the set splits into 12 easy, 20 medium, and 19 hard tasks.

Every task is given the paper, its code repository, and a fresh cloud VM with no pre-installed drivers; Docker is disallowed so that success reflects native dependency resolution. Each task is verified by a hidden pipeline that executes the paper's designated experiment and checks its outputs, and we confirm that all 51 tasks can be deployed by running the reference setup scripts. DeployBench highlights the gap between current agents and autonomous deployment, and offers a realistic testbed for scientific research agents.

After the agent finishes, reaches its step limit, or times out, we evaluate the final VM state with a hidden, task-specific verifier that has two layers. Layer 1 is a global rule-based log parser that catches common execution failures shared across tasks (missing dependencies, runtime exceptions, compilation errors, out-of-memory). If no Layer-1 failure is found, Layer 2 runs task-specific checks for expected outputs, runtime evidence, generated artifacts, or service availability — verifying that the paper-specific target actually produced the required result rather than merely reaching a superficially runnable state. A task is solved only if it passes both layers. For failed tasks, a separate LLM diagnostic judge attributes root causes; all success rates are computed solely from the deterministic verifier.

• Even the strongest agent deploys only ~half of the artifacts. Pass rates span 7.8%–51.0%: GPT-5.3-Codex 51.0%, Gemini-3.1-Pro 27.5%, Grok-4.20 11.8%, and GPT-5.4-Mini 7.8%. End-to-end deployment from raw infrastructure remains hard.
• Failure is mostly a completion-judgment problem. 97 of 154 failed runs are agent-declared self-stops — the agent decides it is done after validating a weaker or different target than the paper-specific task actually requires.
• Generic checks miss most failures. A global rule-based parser catches only 46.1% of failures; the remaining 53.9% pass generic checks and are caught only by per-task verification, motivating DeployBench's two-layer verifier.
• More tokens don't buy success. Gemini-3.1-Pro burns the most tokens (2.55M/task on average) yet ranks second, and within every model failed runs consume more tokens than solved ones.
• Legacy code rots. Legacy repositories (last updated before 2020) far more often need compatibility repair to deploy onto a modern OS and hardware — required in 5 of 10 legacy tasks versus only 3 of 41 recent ones — and every model scores lower on legacy tasks than on recent ones.

Task success rate overall and by research domain, for four frontier LLMs under the OpenHands agent scaffold. Even the strongest model solves only about half of the tasks, and the gap between models shows that DeployBench separates agent capability under a fixed scaffold.

Across the 154 failed runs, the global check (Layer 1) catches 46.1%; the remaining 53.9% reach a superficially runnable state but fail the task-specific verifier (Layer 2) — they would be missed entirely by generic success signals.

Diagnostic-agent reports cluster failures into five recurring root-cause patterns: dependency & package resolution, GPU/CUDA setup, native-build & toolchain, artifact & output mismatch, and system- & VM-level execution. Tasks that all four models fail tend to compound multiple patterns at once.