capture promoted 2026-11-06

Does any outside source treat cross-rater (or cross-model) agreement itself as a diagnostic for distinguishing externally-anchored (popularity-based) from internally-generated (content-based) selection criteria?

wisdom-of-crowds social-influence inter-rater-reliability matthew-effect legitimation popularity bipartite-hop herding

Search direction. The open question (question-does-cross-rater-agreement-serve-as-diagnostic-for-popularity-vs-content-anchored-selection) explicitly pointed away from the two literatures already searched (citation theory; LLM citation-benchmarks) toward "classic inter-rater-reliability literature... for any treatment of agreement-as-diagnostic-for-anchoring, rather than agreement-as-measurement-quality." That redirection paid off: the wisdom-of-crowds / social-influence experimental literature treats exactly this distinction as its central empirical object, under different vocabulary ("social influence effect" vs. "independent judgment," not "popularity-based" vs. "content-based"). A separate search into recent LLM-as-judge bias literature (bandwagon/popularity bias, cross-model ensembling as bias mitigation) surfaced related material but nothing that uses cross-model agreement itself, specifically, as a diagnostic instrument in the way the question asks — that thread is recorded under Further leads, not as a core claim.

Claim: Lorenz, Rauhut, Schweitzer & Helbing (2011) treat convergence of independent raters' estimates as diagnostic of an externally-anchored process specifically because it rises under exposure to others' judgments while leaving accuracy unchanged — the "social influence effect" separates socially-anchored convergence from independent, content-based judgment by comparing against a no-exposure control

verifies: question-does-cross-rater-agreement-serve-as-diagnostic-for-popularity-vs-content-anchored-selection

In a controlled lab experiment (144 ETH Zürich students, real monetary stakes, six factual-estimation questions, five response rounds), the authors compared a "no information" control condition — each subject re-estimates alone, blind to others — against "aggregated" and "full information" conditions where subjects see others' prior estimates before re-estimating. Convergence (reduced diversity across raters) appeared only in the information-exposed conditions, not the control, and — crucially for a diagnostic use of agreement — that convergence did not track any gain in collective accuracy: "The 'social influence effect' diminishes the diversity of the crowd without improvements of its collective error." The paper states its finding directly as the cross-rater-agreement/independent-judgment contrast the question asks about: "knowledge about estimates of others narrows the diversity of opinions to such an extent that it undermines the wisdom of crowd effect in three different ways." Because the no-exposure control preserved diversity (no artificial convergence) while the information-exposed conditions converged without becoming more accurate, the authors use the presence vs. absence of convergence relative to that control — not agreement in isolation — as the empirical signature that distinguishes an externally-anchored process (each rater's estimate pulled toward others' reported values) from an internally-generated one (each rater's own uncontaminated assessment). This is the closest located outside-literature treatment of cross-rater agreement itself functioning as a diagnostic for anchoring vs. independent judgment, though the domain is numerical quantity estimation, not "selection criteria" in the citation/legitimation sense the question was originally posed against.

source_url: "https://sg-dev.github.io/legacy-website/media/publication_files/PNAS-2011-Lorenz-9020-5.pdf" source_title: "How social influence can undermine the wisdom of crowd effect" source_author: "Jan Lorenz, Heiko Rauhut, Frank Schweitzer, Dirk Helbing" source_date: "2011-05-31" source_venue: "Proceedings of the National Academy of Sciences 108(22):9020-9025, DOI 10.1073/pnas.1008636108" source_tier: 1 source_quote: "The "social influence effect" diminishes the diversity of the crowd without improvements of its collective error." source_sha: "6dd8b1618bfff671bcbda7b54496910e57cad7a627e7a357ad9be7e2ed296e12" source_quote_2: "knowledge about estimates of others narrows the diversity of opinions to such an extent that it undermines the wisdom of crowd effect in three different ways" source_url_2: "https://sg-dev.github.io/legacy-website/media/publication_files/PNAS-2011-Lorenz-9020-5.pdf" source_sha_2: "6dd8b1618bfff671bcbda7b54496910e57cad7a627e7a357ad9be7e2ed296e12" source_tier_2: 1 note: "pnas.org's own article pages are known-blocked for this vault's tooling (per 00-meta/specs/sources.md); this is a mirror hosted on the ETH Zürich Chair of Systems Design group's own legacy site (co-author Frank Schweitzer's lab, media/publication_files/ directory of their own past-publications archive) — treated as an author-group-hosted copy, not a third-party scraper, per found-spec's 'cite the author's own site' rule. Both quotes verified via quote_check against the extracted text (sha above); tls verified."

Claim: The same paper explicitly names the confound the question is circling — that in settings where a popularity signal (e.g. a visible download/rating count) is itself part of the social-influence channel, resulting convergence cannot be used to tell whether it tracks real content quality or pure herding, citing Salganik, Dodds & Watts's (2006) music-download experiment as the paradigm case

verifies: question-does-cross-rater-agreement-serve-as-diagnostic-for-popularity-vs-content-anchored-selection

Lorenz et al. raise, and set aside, exactly the ambiguity at the center of the open question — whether observed agreement reflects an externally-anchored (popularity) criterion or an internally-generated (content) one — as a known failure mode of naturalistic "cultural market" settings: "the popularity of a song and its perceived quality emerge through the process of interactive downloading and rating. The herding effects created in this way prevent an objective measurement of quality. Therefore, such settings do not reveal whether social influence works in favor or to the disadvantage of the wisdom of the group." This is the paper's own stated reason for not studying naturalistic popularity-exposed choice settings and instead building a controlled design (monetary incentives tied only to factual accuracy, no benefit to conforming) specifically so that convergence could be interpreted unambiguously as social-influence-driven rather than quality-driven. In other words: an outside source states plainly that raw cross-rater agreement, observed in an environment where popularity signals and content quality co-evolve, is not by itself diagnostic of which one produced it — diagnosing requires a controlled baseline (their own no-information condition) absent in ordinary popularity-exposed settings. Salganik, Dodds & Watts (2006, Science 311:854-856, cited by Lorenz et al. as ref. 19) is the named source for the cultural-market case itself; it was not independently fetched or quoted this session and is flagged below as a further lead, not relied on directly.

Claim: A follow-on theoretical paper (Mavrodiev, Tessone & Schweitzer, 2012) shows that convergence/agreement alone — without knowing the raters' starting diversity and accuracy — cannot be read as a clean diagnostic of externally-anchored vs. internally-generated judgment: the same social-influence process that produces convergence can leave the group more or less accurate depending on where it started

verifies: question-does-cross-rater-agreement-serve-as-diagnostic-for-popularity-vs-content-anchored-selection

Building an agent-based model that reproduces Lorenz et al.'s experimental conditions, the authors report: "we show that the question of whether social influence has a positive or negative net effect on the wisdom of crowds is ill-defined. Instead, it is the starting configuration of the population, in terms of its diversity and accuracy, that directly determines how beneficial social influence actually is." This complicates a naive reading of the first claim above: convergence under social influence always reduces diversity, but whether that convergence also moves the group's aggregate judgment toward or away from truth depends on the population's initial state, not on the presence of convergence itself. For the question at hand, this means cross-rater agreement is diagnostic of an externally-anchored process (social influence acted on these raters) with reasonable robustness, but is not by itself diagnostic of whether that external anchor pulled the group toward or away from a correct, content-grounded answer — a distinct and more modest claim than "agreement proves popularity-anchoring made things worse (or better)."

source_url: "https://arxiv.org/abs/1204.3463" source_title: "Effects of Social Influence on the Wisdom of Crowds" source_author: "Pavlin Mavrodiev, Claudio J. Tessone, Frank Schweitzer" source_date: "2012-04-16" source_venue: "Proceedings, Collective Intelligence 2012 conference; arXiv:1204.3463 [cs.SI]" source_tier: 1 source_quote: "we show that the question of whether social influence has a positive or negative net effect on the wisdom of crowds is ill-defined. Instead, it is the starting configuration of the population, in terms of its diversity and accuracy, that directly determines how beneficial social influence actually is." source_sha: "9a3ee3bbe8e24e4984ed1d380a5eddaa042f4626ee7ac4d00efb4d35784ef079" note: "Fetched via extract_pdf from arxiv.org/pdf; tls verified; quote_check passed. Frank Schweitzer is a co-author of both this paper and the Lorenz et al. 2011 primary above, so this is the same research group extending its own earlier result, not an independent corroboration in the single-source-concentration-cap sense — flagged here as a direct theoretical follow-on, not cited as if it were an unrelated confirming source."

Further leads

Salganik, Dodds & Watts, "Experimental Study of Inequality and Unpredictability in an Artificial Cultural Market" (Science 311:854-856, 2006) — the named source, inside Lorenz et al.'s own citation [19], for the music-download "cultural market" case where popularity and perceived quality become entangled through social influence; not independently fetched this session (science.org is a known-blocked publisher platform per sources.md) — worth a direct read via an author-hosted or arXiv-adjacent copy before promoting any claim about its specific findings.
Banerjee (1992, Quarterly Journal of Economics 107:797-817) and Bikhchandani, Hirshleifer & Welch (1992, Journal of Political Economy 100:992-1026) on herding/informational cascades — cited by Lorenz et al. as the mechanism-lineage for "people may suspect that others have better information" (refs 11-12); treats sequential copying of others' actions (not ratings) as a process that can be uninformative to later observers, a cousin concept to this question's agreement-as-anchoring-signature but framed around decision sequences rather than rater convergence. Not independently fetched or quoted this session — summaries only, via WebSearch, not admissible as sourced claims.
Gao, Chen, Jia, Gong, Lam & Wang, "Evaluating and Mitigating LLM-as-a-judge Bias in Communication Systems" (arXiv:2510.12462, 2026) — names "Bandwagon (Popularity) Bias" as one of 11 tested LLM-judge biases and proposes a "jury of models" (cross-provider ensemble) as a mitigation on the assumption that biases are uncorrelated across judges — this treats cross-model agreement as a bias-dilution mechanism, not as a diagnostic for distinguishing popularity-anchored from content-anchored verdicts; the paper does not test whether agreement rates differ between bandwagon-biased and clean conditions, so it does not answer the core question as posed. Fetched and read this session (sha 25f85fbcafd84daed853092a9a98f197eada208e34ba6940ef5a7e75693cfb24) but not used as a claim since its design doesn't operationalize agreement-as-diagnostic.
Marioriyad, Rohban & Baghshah, "The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge" (arXiv:2509.26072, 2025) — injects provenance/recency labels into pairwise LLM-judge comparisons and measures per-model verdict-shift rates (GPT-4o, Gemini-2.5-Flash); close to this question's territory (bias diagnostics for LLM judges) but measures single-judge shortcut sensitivity, not cross-model agreement itself, and was not independently fetched/quoted this session (WebSearch summary only) — a candidate for direct reading in a follow-up session.
Rauhut & Lorenz's 2011 reply to Farrell in the same PNAS volume (E626) — a secondary exchange, already fetched this session, confirming the "collective tunnel vision" reading of the social influence effect but adding no new diagnostic claim beyond the primary paper.

Entity candidates

Francis Galton — person — the foundational figure the whole wisdom-of-crowds tradition (including Lorenz et al.) measures itself against: Lorenz et al. open by citing Galton's 1907 Nature note "Vox populi" as the original demonstration that a group median can beat individual experts, and frame their own paper as showing the conditions under which that 101-year-old result breaks down. Flagged first per the vault's known blind spot for under-flagging the older figure a comparison rests on.
James Surowiecki — person — popularized "the wisdom of crowds" as a named concept (2004 book, cited by both primaries here); the secondary popularizing figure between Galton and the current experimental literature.
Jan Lorenz, Heiko Rauhut, Frank Schweitzer, Dirk Helbing — persons — authors of the 2011 PNAS primary; Schweitzer also co-authors the 2012 follow-on model paper.
Matthew J. Salganik, Peter S. Dodds, Duncan J. Watts — persons — authors of the 2006 Science cultural-market study that Lorenz et al. cite as the paradigm case where popularity and quality cannot be disentangled from observed convergence; not yet independently read in this vault.
Sushil Bikhchandani, David Hirshleifer, Ivo Welch; Abhijit Banerjee — persons — originators of the informational-cascade/herding-model literature (1992), the economics-side cousin tradition to the social-influence/wisdom-of-crowds literature, cited by Lorenz et al. but not independently fetched this session.
wisdom of crowds — concept — candidate for its own entity/concept page; distinct from but adjacent to entity-cross-validation and entity-dawid-skene-model already surfaced by this topic's retrieval.
diversity prediction theorem (Scott E. Page, 2007) — concept — the mathematical identity (collective error = average individual error − group diversity) both primaries use to formalize why reduced diversity under social influence is informative; could support its own note.

written by claude-sonnet-5 · Batch research run, 2026-11-06, targeted against [[question-does-cross-rater-agreement-serve-as-diagnostic-for-popularity-vs-content-anchored-selection]] · raw markdown