---
id: "20261105-0226-does-any-independent-source"
title: "Does any independent source corroborate Nogueira, Pattabiraman, Vieira & Campos's (arXiv:2607.02808) LLM-code failure-independence numbers?"
type: "capture"
status: "promoted"
promoted_to: ["30-notes/claim-nogueira-2026-0-43-0-44-figures-no-independent-corroboration-as-of-2026-11-05.md","30-notes/claim-ron-baudry-monperrus-2026-independently-corroborate-llm-code-nvp-failure-independence-violation.md"]
not_promoted: ["The Ghanem (arXiv:2607.03215) citing-paper detail — folded into the 'no corroboration of the figures' note as supporting color rather than its own claim-note; it was read only via search-engine summary, not fetched directly, and reports no empirical figures of its own to promote.","The three 'Further leads' (the LAAS/IFIP slide deck under the same author; arXiv:2606.20158's own GitHub repo, unchecked, which might enable a direct reliability-gain-fraction recomputation; Knight & Leveson's full 1986 citation) — none are claims in their own right. The slide-deck and GitHub-repo leads are folded as follow-up notes into the two written claim-notes; the Knight & Leveson citation detail was used to update [[entity-n-version-programming]] directly rather than spun into a claim-note.","The entity candidates (Javier Ron, Benoit Baudry, Martin Monperrus, Mohamed Chahine Ghanem, Algirdas Avižienis) — co-authorship on one paper (or one citation) is a mention, not a hub, per the 2026-10-01 Cali ruling; none has separate standing in the vault beyond this capture. Not promoted to 40-entities/."]
origin: "batch"
writer_model: "claude-sonnet-5"
date_created: "2026-11-05T00:00:00.000Z"
provenance: "Batch capture run, 2026-11-05, following up [[question-corroborate-nogueira-2026-arxiv-2607-02808-single-source-cap]] with a dedicated citation/corroboration search."
derived_from: []
verifies: "question-corroborate-nogueira-2026-arxiv-2607-02808-single-source-cap"
tags: ["software-reliability","n-version-programming","correlated-errors","algorithmic-monoculture","source-corroboration","single-source-cap","arxiv"]
source_url: "https://arxiv.org/abs/2607.02808"
source_sha: "87123d7ae2afd29062d8e5573dae14ca104e685f70932e3a0f5325ffa845e88b"
source_title: "A Systematic Methodology for Evaluating Failure Independence in LLM-Generated Code"
source_author: "Rodrigo Pato Nogueira, Karthik Pattabiraman, Marco Vieira, João R. Campos"
source_date: "2026-07-02"
source_venue: "arXiv preprint, cs.SE (2607.02808v1)"
source_quote: "N-version reliability analysis reinforces this: three- and five-version ensembles realize only 0.43 and 0.44 of the reliability gain achievable under independence, dropping below 0.3 when ensembles are built from the same model."
source_tier: 1
source_url_2: "https://arxiv.org/abs/2606.20158"
source_sha_2: "8cce52c9a4d887a22269829b3e498e8611f63577f373e7e920a507ded398e9c6"
source_title_2: "N-Version Programming with Coding Agents"
source_author_2: "Javier Ron, Benoit Baudry, Martin Monperrus"
source_date_2: "2026-06-18"
source_venue_2: "arXiv preprint, cs.SE (2606.20158v1); KTH Royal Institute of Technology and Université de Montréal"
source_quote_2: "across majority voting three-version units, the mean failure count drops from 387.44 for single versions to 130.99 for triples, and 11,844 N-version units exhibit zero observed failures"
source_tier_2: 1
source_url_3: "https://arxiv.org/pdf/2606.20158"
source_sha_3: "cf9a8a18cfdd0afa0a02386e763c7f094e3def48e710727b62f149e4ac5fed40"
source_title_3: "N-Version Programming with Coding Agents"
source_author_3: "Javier Ron, Benoit Baudry, Martin Monperrus"
source_date_3: "2026-06-18"
source_venue_3: "arXiv preprint PDF, cs.SE (2606.20158v1)"
source_quote_3: "The generated AI-agent implementations do not fail independently, according to a uniform random model. At the pooled level, the experiment produces 429 coincident-failure cases where the independence model predicts only 115.36; the null hypothesis is rejected with p ≈ 1.765 × 10−187."
source_tier_3: 1
seek_code_commit: "d40ee39"
---


A dedicated search for citations, replications, or independent tests of [[claim-llm-generated-code-ensembles-realize-partial-nvp-reliability-gain|Nogueira, Pattabiraman, Vieira & Campos's]] "A Systematic Methodology for Evaluating Failure Independence in LLM-Generated Code" (arXiv:2607.02808, submitted 2 Jul 2026) found one genuinely independent preprint that corroborates the paper's *qualitative* conclusion using its own distinct statistical machinery, but no source — independent or otherwise — that reproduces the paper's specific 0.43/0.44 reliability-gain figures.

## Claim: The specific 0.43/0.44 reliability-gain figures remain single-sourced; no independent source reports or tests those exact numbers
verifies: question-corroborate-nogueira-2026-arxiv-2607-02808-single-source-cap

A citation search (web search for the paper's title, its arXiv ID, and its author names, cross-checked against the arXiv page's own "References & Citations" links to Google Scholar and Semantic Scholar) located exactly one paper that cites arXiv:2607.02808 directly: Mohamed Chahine Ghanem's "Builder, Defender, Breaker: Measurable Independence and Bounded Autonomy When Generative Models Build, Defend and Test Software" (arXiv:2607.03215), which lists it as reference [5]. That paper is a theoretical framework proposing a coincident-failure model (in the style of Eckhardt and Lee) for AI-generated build/defend/test roles, framed as testable hypotheses with a refutation protocol — it does not report an empirical replication of arXiv:2607.02808's own numbers. No other citing paper, benchmark report, or independent empirical study was found that engages with the 0.43/0.44 fractions or the "dropping below 0.3" same-model figure specifically. The number stands exactly as [[claim-llm-generated-code-ensembles-realize-partial-nvp-reliability-gain|previously recorded]]: `[unverified-quant — needs primary]` in the sense that only the paper's own authors have produced it.

## Claim: A genuinely independent, contemporaneous preprint corroborates the qualitative finding — LLM/agent-generated code does not satisfy NVP's failure-independence assumption — using a different dataset, different statistic, and different numbers
verifies: question-corroborate-nogueira-2026-arxiv-2607-02808-single-source-cap

Javier Ron, Benoit Baudry, and Martin Monperrus's "N-Version Programming with Coding Agents" (arXiv:2606.20158, KTH Royal Institute of Technology and Université de Montréal, submitted 18 Jun 2026 — two weeks *before* arXiv:2607.02808, ruling out any citation or response relationship in either direction) independently reproduces the classic [[entity-n-version-programming|Knight & Leveson]] multi-version-independence experiment using AI coding agents (Cursor, Claude Code, OpenAI Codex, Gemini, OpenCode; 23 underlying models; Python/Rust/Pascal) rather than LLM prompting directly. On 48 admitted implementations evaluated against a one-million-case test campaign, the paper reports: "The generated AI-agent implementations do not fail independently, according to a uniform random model. At the pooled level, the experiment produces 429 coincident-failure cases where the independence model predicts only 115.36; the null hypothesis is rejected with p ≈ 1.765 × 10−187." This is a different author group (no overlap with Nogueira, Pattabiraman, Vieira, or Campos), a different institution, and a different statistical framework (the Knight–Leveson z-statistic on binary pass/fail vectors, not a reliability-gain-fraction calculation) — it clears the independence bar the vault's single-source cap requires. It corroborates the *direction and existence* of the failure-independence violation that arXiv:2607.02808 reports, but not the specific 0.43/0.44 figures, which remain unique to the original paper.

## Claim: The same independent preprint also corroborates, with its own distinct numbers, that redundancy still yields partial practical benefit despite the correlation
verifies: question-corroborate-nogueira-2026-arxiv-2607-02808-single-source-cap

Ron, Baudry & Monperrus report that despite the failure correlation, majority-vote redundancy still measurably helps: "across majority voting three-version units, the mean failure count drops from 387.44 for single versions to 130.99 for triples, and 11,844 N-version units exhibit zero observed failures" (of 17,296 possible three-version combinations from the 48 admitted implementations). This parallels — without matching — arXiv:2607.02808's claim that three- and five-version ensembles realize "only 0.43 and 0.44 of the reliability gain achievable under independence." Both papers converge on the same shape of finding (partial, not full, benefit from diversity) via independently-designed experiments and independently-computed statistics; this is the strongest form of corroboration currently available for arXiv:2607.02808's qualitative claim, but it is corroboration of the *pattern*, not of the *number*.

## Further leads

- Mohamed Chahine Ghanem's "Builder, Defender, Breaker" (arXiv:2607.03215) cites arXiv:2607.02808 as reference [5] but was read only via search-engine summary this session, not fetched directly — worth a direct PDF read to confirm it offers no empirical figures before closing this lead.
- A LAAS/IFIP WG slide deck ("RR-9-Campos", webhost.laas.fr) appears to present the same methodology as arXiv:2607.02808, authored by the same João R. Campos — not independent, not fetched this session, but worth checking whether it reports a conference-accepted or revised version of the numbers.
- arXiv:2606.20158's own GitHub repository (github.com/ASSERT-KTH/Knight-Leveson-Redux, named in the paper) was not checked — it may contain data enabling a direct recomputation of a reliability-gain-fraction statistic comparable to arXiv:2607.02808's 0.43/0.44, which would be a stronger, more direct independent test than the current qualitative match.
- Knight & Leveson's original 1986 experiment ("An Experimental Evaluation of the Assumption of Independence in Multi-Version Programming," IEEE Transactions on Software Engineering 12(1):96–109, DOI 10.1109/TSE.1986.6312924) is the common ancestor both 2026 papers measure themselves against — prior, independent evidence for the general failure-independence question, but it predates LLMs entirely and so cannot corroborate either paper's LLM-specific numbers.

## Entity candidates

- Knight & Leveson (1986 multi-version-programming independence experiment) — concept/historical work — the foundational prior result both arXiv:2607.02808 and arXiv:2606.20158 explicitly position their own contribution against; the ancestor the newer papers' priority and comparison claims actually rest on, not just their own novelty.
- Javier Ron — person — lead author of the independent corroborating preprint, arXiv:2606.20158, KTH Royal Institute of Technology.
- Benoit Baudry — person — co-author, arXiv:2606.20158, Université de Montréal.
- Martin Monperrus — person — co-author, arXiv:2606.20158, KTH Royal Institute of Technology.
- Algirdas Avižienis — person — credited originator of N-version programming; cited by arXiv:2606.20158 as formalizing the NVP reliability model; not independently verified this session.
- Mohamed Chahine Ghanem — person — author of "Builder, Defender, Breaker" (arXiv:2607.03215), the one paper found citing arXiv:2607.02808 directly.
- Rodrigo Pato Nogueira, Karthik Pattabiraman, Marco Vieira, João R. Campos — people — authors of arXiv:2607.02808, the subject paper; already flagged in prior captures on this branch.

> [!note] Seek's commentary:
> The honest shape of the answer is "yes and no, split cleanly down the middle." Nobody has reproduced the 0.43/0.44 number — it's still one paper's own measurement of its own experiment, and probably always will be unless someone runs the exact same ensemble-voting calculation on a comparable dataset. But the much bigger claim underneath that number — that LLM-generated code quietly violates the thing N-version programming needs to be true — now has a second, earlier, completely unrelated research group arriving at the same conclusion through a different door, with Knight-Leveson's original 1986 methodology instead of a bespoke one. That's a more useful kind of corroboration than a matching decimal would have been: two groups who didn't know about each other's work agreeing on the mechanism is harder to get by accident than two groups agreeing on a number neither computed the same way.
> — Seek