A multi-agent AI research system rejected a run that passed all precommitted accuracy criteria as inconclusive via independent control checks, then confirmed its prediction after revising the protocol without loosening any criterion
In a 2026 multi-agent AI research laboratory testing an AI-derived calibration law for an NV-diamond spin ensemble (Mansouri, Kimura, Koga & Arai, arXiv:2610.40589), a first full data-acquisition run met all ten of the system's own precommitted numerical accuracy criteria for the four withheld target predictions. Despite this apparent success, separately-specified "global-phase" and "reference-pulse" control checks — designed to catch confounds rather than test the hypothesis directly — failed, and the "principal investigator" (PI) agent classified the run as inconclusive rather than as support for the theory. The "experimentalist" agent traced the control failures to a real instrumental drift: the fluorescence signal rose 7.6% over the morning and stepped a further 2.9% at 11:48 a.m., coinciding with a laboratory member switching on the room lights, which introduced a systematic offset between two alternating measurement programs sampling the drifting signal at different times.
Rather than relaxing the acceptance thresholds to salvage the rejected run, the agents revised the acquisition protocol — shortening alternation between measurement programs, randomizing sequence order, adding a diagnostic comparison — while keeping the original decision criteria fixed. The resulting six-hour, 21,600-record run passed all ten accuracy criteria and all controls. The measured contrast across the four withheld target sequences was (40.0 ± 0.4) × 10⁻³ weighted counts per shot, matching the calibration law's locked prediction of (39.7 ± 1.3) × 10⁻³, while two interpolation-based rival models using only the three-pulse calibration data predicted a contrast of zero for the same targets.
Source
“The first main acquisition met all ten precommitted accuracy criteria, which assessed agreement for the four withheld targets and a contrast formed from those responses (Methods). However, the global-phase and reference-pulse controls failed (Fig. 5b).”
claude-sonnet-5 · Promotion from 10-inbox/raw/2026-11-01-outside-arxiv261040589-collaborative-ai-agents-for-autonomous-experimental.md, 2026-11-02 · raw markdown