Lorenz, Rauhut, Schweitzer & Helbing (2011) show cross-rater convergence is diagnostic of social-influence (externally-anchored) judgment only against a no-information baseline — not from agreement alone in naturalistic, popularity-exposed settings

wisdom-of-crowdssocial-influenceinter-rater-reliabilitymatthew-effectlegitimationpopularitybipartite-hopherding

In a controlled lab experiment (144 ETH Zürich students, real monetary stakes, six factual-estimation questions, five response rounds), the authors compared a "no information" control — each subject re-estimates alone, blind to others — against "aggregated" and "full information" conditions where subjects see others' prior estimates before re-estimating. Convergence (reduced diversity across raters) appeared only in the information-exposed conditions, not the control, and that convergence did not track any gain in collective accuracy: "The 'social influence effect' diminishes the diversity of the crowd without improvements of its collective error." The paper's diagnostic move is comparative, not absolute: it is the presence of convergence relative to the no-exposure control — not agreement observed in isolation — that marks a process as externally-anchored (each rater's estimate pulled toward others' reported values) rather than internally-generated (each rater's own uncontaminated assessment).

The same paper names, and explicitly sets aside, the case where this diagnostic breaks: naturalistic "cultural market" settings where a popularity signal (e.g. a visible download or rating count) co-evolves with the judgment itself. There, "the popularity of a song and its perceived quality emerge through the process of interactive downloading and rating. The herding effects created in this way prevent an objective measurement of quality. Therefore, such settings do not reveal whether social influence works in favor or to the disadvantage of the wisdom of the group." This is the paper's own stated reason for building a controlled design with monetary incentives tied only to factual accuracy — so that convergence could be read unambiguously as social-influence-driven rather than quality-driven, a baseline ordinary popularity-exposed settings lack.

For question-does-cross-rater-agreement-serve-as-diagnostic-for-popularity-vs-content-anchored-selection, this is the closest located outside-literature treatment of cross-rater agreement itself functioning as a diagnostic for externally-anchored vs. independent judgment — though the domain is numerical quantity estimation under social influence, not "selection criteria" in the citation/legitimation sense (entity-citation-theory-normative-vs-social-constructivist) the question was originally posed against, and the diagnostic requires a controlled baseline absent from the citation-selection setting.