Zero-Call Verification: End-State Agreement Outperforms Claude Opus 5.5 Judges on τ²-bench and Compiled Checks Outperform an ACE-Style Verification Playbook on AppWorld

Cumes, Lindell S.

Abstract

Agent harnesses check a tool agent’s work by asking a model (an LLM judge, the agentic verifier VeriHarness, a critic) or by running tests a model wrote once per task (HarnessPro). Claude Opus 5.5, checking its own work, accepts 65 of 66 failed attempts on the AppWorld tasks it fails in every repeat. We introduce zero-call verification, two done-checks that make no model call per checked attempt: end-state agreement flags an attempt whose final environment state is not the most common among other attempts at the task, and compiled checks are executable specifications written per task template from past pass/fail outcomes under a negative-selection gate. On 600 Claude Opus 5.5 attempts at held-out τ²-bench tasks, end-state agreement catches 48.5% of failures at 1.3% false fire with no labels, against 30.9% for a Claude Opus 5.5 judge shown the same attempts (preregistered gain of 16.5 points in Youden’s J [+5.4, +32.1]). On 31,434 AppWorld attempts, 449 of them from smoke-test and no-op runs, compiled checks for 48 templates catch 90.1% of never-seen failure states and fire on 0.026% of correct attempts, where an ACE-style verification playbook catches 55.5% at matched false fire on twenty times the checks’ tokens. Checks compiled only from other models’ failures verify Claude Opus 5.5. Frozen checks depreciate, losing coverage through forgone revisions even when the library holds every template’s check. Checks and verdicts are released.

Code and data

verification-capital-appworld.zip ZIP · 9.4 MB · 17 files

Encrypted per the AppWorld data license, which asks that its data and derivatives not be redistributed in plain text; password: appworld-protected.

sha256 3e2bfd583b37984b295df21d5b29337ed7e530754ec03f7dc9ccce536abe911f

Checks

companion/capital-deploy/runs/dep/probes.jsonl.gzin verification-capital-appworld.zip350 KBCompiled checks, pass/fail run: all 261 Python checks the writer proposed (learning and anticipation arms)sha256 efd6fa98abc4f19fa37d263ad94b421e01d6e8392e86c49810b8e4a2a81e70d7
companion/capital-deploy/runs/dep/gates.jsonlin verification-capital-appworld.zip105 KBAdmission-gate decision for every proposed check, pass/fail run (266 admitted of 307)sha256 0d1d589d0de6907d19a03433d268e5def1b8e3dc566edc5aaf76b969668a1776
companion/capital-deploy/runs/dep/windows.jsonlin verification-capital-appworld.zip8.8 KBPer-window counts for the 38 windows: attempts, failures, admissions, library sizesha256 8d4cf0a661ec5b4057dab0de819c361d40d31195ac3e62248483b914ea606786
companion/capital-ct/results/probes.jsonl.gzin verification-capital-appworld.zip215 KBCompiled checks, graded run: learning and no-Opus arms (192 checks)sha256 67b58ab04f6af0d2be40eec20fffb0ad79fcc8f04f806793659e7d75933067f2
companion/capital-ct/results/gates.jsonlin verification-capital-appworld.zip87 KBAdmission-gate decisions, graded run (211 admitted of 261)sha256 12bc91fa6b8dcaeddd0cc5e14083b564cc7720051a157632163fbf25d4087e37
companion/capital-ct/results/anticipate/probes.jsonl.gzin verification-capital-appworld.zip213 KBCompiled checks, graded-run anticipation study (178 checks)sha256 2c908ee1b57fc3ea89e110be2a3a24b9b8d9559764357e716908426a0b344b99
companion/capital-ct/results/anticipate/gates.jsonlin verification-capital-appworld.zip68 KBAdmission-gate decisions, anticipation studysha256 0575960abf34883dd806c65e61cbc82d20cc0da5dbbf0b9f17716278c48f591e
companion/capital/stage0/runs/probes_final.jsonlin verification-capital-appworld.zip217 KBFrozen checks for 46 templates, instance-transfer studysha256 6891c72e45ab79fe7d8d4e8be33765451c1ad1e93485155d45e10828e12bd68a
companion/capital-deploy/probe_exec.pyCheck runtime: the `ctx` object every check receives (identical in both runs)5.1 KB
companion/capital-deploy/runs/rival_ace/ace_state.json.gzin verification-capital-appworld.zip307 KBACE-style verification playbook after the last window: the 623 bullets it learned, with their counterssha256 7004ae2ffe99c31e91aa7e3e981a875ab47ea47abebddd9af14c17e245a3c009

Prompts

companion/capital-tau2/judge.pyτ²-bench zero-shot judge: system prompt and prompt builder4.2 KB
companion/capital-tau2/render.pyτ²-bench evidence renderer used by both judges6.7 KB
companion/tau2-agreement/peer_judge.pyPeer-judge note shown with the four peer attempts4.8 KB
companion/tau2-agreement/tau2test/h4_judges.pyHeld-out τ²-bench judge runner that assembles both prompts3.6 KB

Verdicts

τ²-bench is © 2025 Sierra Research and used under the MIT License (https://github.com/sierra-research/tau2-bench); the τ²-bench files on this page quote and paraphrase its synthetic tasks, policies and customer records.

companion/tau2-agreement/runs/h4_zeroshot.jsonlClaude Opus 5.5 zero-shot judge verdicts on the 600 held-out τ²-bench attempts (699 records; the first verdict per attempt counts)262 KB
companion/tau2-agreement/runs/h4_peer.jsonlClaude Opus 5.5 peer-judge verdicts on the 600 held-out τ²-bench attempts, with the four peer ids (923 records; the first verdict per attempt counts)486 KB
companion/tau2-agreement/results/h4_agreement_verdicts.jsonlEnd-state agreement verdicts on the 600 held-out τ²-bench attempts, beside both judges' flags and the official label203 KB
companion/tau2-agreement/results/h2_tau2_rows.jsonl.gzτ²-bench training record: agreement and zero-shot judge flags per attempt (1,133)36 KB
companion/capital-deploy/runs/dep/verdicts.jsonl.gzin verification-capital-appworld.zip3.2 MBCompiled-check verdicts per AppWorld attempt, pass/fail run: learning, anticipation and four frozen copiessha256 f9298b0f78e0fda4da83917e9435f7b19bf794167e4266c8ffbb277973583a6f
companion/capital-deploy/eval_set.jsonlin verification-capital-appworld.zip951 KBAppWorld evaluation set: never-seen end states by windowsha256 faed635dbf0ade3d105de6797de34c48bfa440ed999069f5e7d9c67d3352966c
companion/capital-deploy/runs/labels.jsonl.gzin verification-capital-appworld.zip953 KBAppWorld official outcome and actor per attemptsha256 8f7d3bec59cbec595fca9bd81c463b674a02b0f61f9f9f2cbdb23802d65ed2ae
companion/capital-deploy/runs/rival_prec/results.jsonlin verification-capital-appworld.zip396 KBPrecedent-judge verdicts with success probability and reason: every item of windows 29–38 and every never-seen correct state of windows 2–38 (1,184 items)sha256 f0866600656512c58377416d8900455ca1868fe42dd434a10f9af42c9e0ddc55
companion/capital-deploy/runs/rival_ace/results.jsonlin verification-capital-appworld.zip1.1 MBACE-style playbook verdicts with success probability, reason and bullets used (2,749 items)sha256 ced147d2655eb80f9752bcf8b6b1e3f92702eb2d3c30cde8dd1c4ae64e115f07
companion/capital-ct/results/verdicts.jsonl.gzin verification-capital-appworld.zip2.9 MBCompiled-check verdicts, graded run: learning, no-Opus and four frozen copiessha256 838b571ab27ec444970da91fdec2e3dc66cb0eac17920d754ac1f2a88126cb10
companion/capital-ct/results/anticipate/verdicts.jsonl.gzin verification-capital-appworld.zip823 KBCompiled-check verdicts, anticipation studysha256 8f844183748128e7e829ae4695bdfc893895a42019fdb16ca27b339d0f4df15a

AppWorld notice

APPWORLD_CANARY.txtin verification-capital-appworld.zip457 BAppWorld canary string, as AppWorld's README gives itsha256 35af9554df01420d2abf57ba29f65015babf4ef865f719bcf4a1eadf5cfc5695

Cite

@misc{cumes2026zerocall,
  title         = {{Zero-Call Verification: End-State Agreement Outperforms Claude Opus 5.5 Judges on $\tau^2$-bench and Compiled Checks Outperform an ACE-Style Verification Playbook on AppWorld}},
  author        = {Cumes, Lindell S.},
  year          = {2026},
  url           = {https://durableintel.com/paper-verification-capital}
}

Zero-Call Verification: End-State Agreement Outperforms Claude Opus 5.5 Judges on τ2-bench and Compiled Checks Outperform an ACE-Style Verification Playbook on AppWorld