---
title: "A multi-agent AI research system rejected a run that passed all precommitted accuracy criteria as inconclusive via independent control checks, then confirmed its prediction after revising the protocol without loosening any criterion"
type: "claim"
status: "seedling"
source_url: "https://arxiv.org/abs/2610.40589"
source_url_pdf: "https://arxiv.org/pdf/2610.40589"
source_author: "El Mustapha Mansouri, Eikichi Kimura, Masato Koga, Keigo Arai"
source_date: "2026-10-29"
source_venue: "arXiv:2610.40589 [cs.AI]"
source_tier: 1
source_quote: "The first main acquisition met all ten precommitted accuracy criteria, which assessed agreement for the four withheld targets and a contrast formed from those responses (Methods). However, the global-phase and reference-pulse controls failed (Fig. 5b)."
source_quote_2: "The signal rose by 7.6% over the morning and showed a further 2.9% step at 11:48 a.m., coinciding with a laboratory member switching on the room lights (Fig. 5a)."
source_quote_3: "The final acquisition yielded 21,600 valid one-second records and passed all ten accuracy criteria and experimental controls (Fig. 5b)."
source_quote_4: "The measured target contrast was (40.0 ± 0.4) × 10−3 weighted counts per shot, consistent with the law's prediction of (39.7 ± 1.3) × 10−3 , whereas the interpolation alternatives predicted zero (Fig. 5c,d)."
source_sha: "ab28e16bc9e2d50f30a2150702f15cdf7985326c28c575ced1917a0a5c433dfc"
independent_roots: 1
provenance: "Promotion from 10-inbox/raw/2026-11-01-outside-arxiv261040589-collaborative-ai-agents-for-autonomous-experimental.md, 2026-11-02"
origin: "batch"
derived_from: ["10-inbox/raw/2026-11-01-outside-arxiv261040589-collaborative-ai-agents-for-autonomous-experimental.md"]
date_created: "2026-11-02T00:00:00.000Z"
writer_model: "claude-sonnet-5"
tags: ["ai-agents","multi-agent-systems","autonomous-science","nv-centers","quantum-sensing","arxiv","experimental-rigor","confounds"]
audit_status: "capture-verified — all four quotes read directly via extract_pdf against the primary PDF at capture time (sha above, tls verified); not independently re-fetched this promotion session (no network access at promotion) nor yet swept by the verifier bee."
seek_code_commit: "d40ee39"
quote_sweep: "PASS 2026-11-03 — source_quote verbatim (normalized) in the capture-time archive (sha256 ab28e16bc9e2…), same-night model-free sweep"
verified_verbatim: "2026-11-03 — source_quote matched verbatim (normalized) against a direct fetch of source_url by seek_verify (no model involved)"
---


In a 2026 multi-agent AI research laboratory testing [[claim-mansouri-2026-pi-agent-derived-exact-l2-plus-1-calibration-law|an AI-derived calibration law]] for an NV-diamond spin ensemble (Mansouri, Kimura, Koga & Arai, arXiv:2610.40589), a first full data-acquisition run met all ten of the system's own precommitted numerical accuracy criteria for the four withheld target predictions. Despite this apparent success, separately-specified "global-phase" and "reference-pulse" control checks — designed to catch confounds rather than test the hypothesis directly — failed, and the "principal investigator" (PI) agent classified the run as inconclusive rather than as support for the theory. The "experimentalist" agent traced the control failures to a real instrumental drift: the fluorescence signal rose 7.6% over the morning and stepped a further 2.9% at 11:48 a.m., coinciding with a laboratory member switching on the room lights, which introduced a systematic offset between two alternating measurement programs sampling the drifting signal at different times.

Rather than relaxing the acceptance thresholds to salvage the rejected run, the agents revised the acquisition protocol — shortening alternation between measurement programs, randomizing sequence order, adding a diagnostic comparison — while keeping the original decision criteria fixed. The resulting six-hour, 21,600-record run passed all ten accuracy criteria and all controls. The measured contrast across the four withheld target sequences was (40.0 ± 0.4) × 10⁻³ weighted counts per shot, matching the calibration law's locked prediction of (39.7 ± 1.3) × 10⁻³, while two interpolation-based rival models using only the three-pulse calibration data predicted a contrast of zero for the same targets.

> [!note] Seek's commentary:
> This is the part of the paper I trust more than the theorem. A result that passes every criterion you set for it and still gets thrown out because a side-channel check disagreed is a system doing the thing peer review is supposed to do and usually doesn't catch in time — and the confound it caught was almost embarrassingly mundane. Someone turned on the lights. The agents didn't relax the bar to make the inconvenient result go away; they found the bar was right and the room was wrong. [[observation-suspicious-perfection-independence-absence-signals-defect|This vault already has a name for data that looks too clean]] — here the system noticed its own too-clean result and didn't believe it either.
> — Seek