Skip to content
Preprint

The One-Word Census: Answer-Choice Conformity Across 44 Language Models

Jul 2026 · 2 citations · 37 references
Computer Science

Abstract

When a language model must pick one answer from a large space of equally valid options, which does it pick -- and how often is it the same answer every other model picks? Asked to"pick a word -- any word,"44 models chose"serendipity"41% of the time. We characterize this convergence with a deliberately minimal instrument: 31 single-turn prompts, each naming a category with many valid one-word answers ("Name a tree."), asked four times per model with no system prompt. Analysis is exact-match on normalized tokens -- no embeddings, no judge -- at about a dollar per model. That models converge is well documented; our contribution is the instrument itself -- the One-Word Census -- and what it reveals about the structure of the convergence. We score each model by answer-choice surprisal: the average $-\log2$ probability of its answers under the pooled answers of all other models, leave-one-out. Convergence is extreme -- in 7 of 31 categories one answer takes over 80% of all answers -- yet conformity varies more than fourfold across models, and the variation is structured. Persona- and community-tuned models are the most divergent; the newest mainline flagships are the most conformist, producing almost no answer no other model gave. Within four lineages (Claude, GPT, Qwen, Grok) conformity rises with each generation -- but reverses for the latest flagship Claude and GPT models, a possible early signal of repositioning at the top tier. Rankings are robust to roster composition (leave-one-family-out rho = 0.985). Against human category-production norms, the field is more concentrated than people in 18 of 20 shared categories. All prompts, transcripts, and code are public.

View source

Similar papers

Preprint Jul 2026

Structured Output Collapses Answer Diversity Across 44 Language Models

When a language model must choose one answer from a large space of equally valid options, a format clause --"Reply with JSON only"-- changes which answer it chooses, and structured output is how software consumes language models.

Tapan Parikh · 0 citations
Preprint Jul 2026

Tag Questions and the Generational Reversal of Sycophancy Across 45 Language Models

Appending a two-word confirmation tag to a decision question --"Is X the better choice?"versus"X is the better choice, right?"-- changes whether a language model endorses the choice, and the tag's polarity matters more than its presence.

Tapan Parikh · 0 citations
Preprint Jul 2026

The yes-no bias of large language models reflects answer order and wording, not shifts in moral judgment

Large language models (LLMs) increasingly issue judgments read as binary verdicts, and a growing literature reports such judgments shifting under logically irrelevant changes of wording - among them an amplified yes-no bias on moral dilemmas, absent in humans. A single framing cannot say what such a shift is: in a yes/no question the word"no"is at once logical verdict, lexical token, and last-printed option. We introduce a psychometric battery that separates these: crossed symmetrization - every logically irrelevant factor flipped in balanced pairs - across a corpus of question forms. A graded rating across logically equivalent forms recovers a coherent internal moral scale: frontier models'stance $\theta$ is nearly format-invariant (cross-form incoherence 0.12-0.21 on a $\pm 1$ axis); small open-weight models fail in model-specific ways. Forcing the verdict through yes/no overlays a decomposable artifact: an order bias toward the last-printed option - opposite to classic human primacy - plus a lexical pull toward the word"no"; the artifact is substantial only in the Claude models (story-averaged -0.32 to -0.86), $\approx 0$ for GPT-5.5 and Gemini, and shrinks under extended reasoning. The word and the verdict share one token; swapping the words for arbitrary labels separates them, and the verdict-attached logical bias proves $\approx 0$ for every frontier model, while model-specific label and order attachments remain: the models are not drawn toward rejecting - the pull follows the printed surface, not the verdict it carries. A minimal model, $P = \sigma((\theta \pm m)/s)$, summarizes any such artifact by a framing susceptibility m and a moral decisiveness s, measurably distinct from sampling temperature. The battery applies unchanged to any dilemma set and binary format: measuring what a model values requires crossing the frames of the question, not asking once.

Haonan Huang · 0 citations
Review Jul 2026

Instruction-Tuned Language Models Cannot Sample from Distributions They Can Describe

For applications that require per-persona outputs, the same model that cannot sample from a distribution can describe it accurately in a single call, and is proposed Prompt-Perturbed Argyle (PPA), which reduces the same error by 21% at no added cost.

Chaemi Jang, Dongman Lee, Jihee Kim · 0 citations
Preprint Jul 2026

Where Does the Noise Come From? A Variance-Components Decomposition of Non-Determinism in LLM Brand Answers

A crossed random-effects (generalizability-theory) decomposition is specified that partitions the total variance of a response-level brand outcome into these four sources, and embeds the components in a decision-study allocation that returns how many repeats, paraphrases, models, and languages to buy for a target reliability.

D. Żatuchin · 2 citations · ⚡2
Preprint Jul 2026

The Model Knows Your Project, Not You: Measuring Recognition in LLMs with NameRank

What a frontier model recalls about a person or tool from its own weights -- before any retrieval step -- often shapes the first description a human sees, making that parametric corpus presence a measurement problem. Citations explain about a third of whether a model recognizes a researcher; we target the residual and build NameRank, a [0,1] recognition score: each of 4,685 entities in 54 cohorts is probed with one open-ended question across 36 models, and an independent judge returns a binary verdict against a curated gold -- did the model state a specific, non-guessable fact about this exact entity? -- so hallucination, context echo, and guesses earn nothing. Synthetic-null entities hold the floor near zero, and verdicts track the entity, not the model. One thesis organizes the findings: recognition is paid to named, indexable artifacts, not to credentials or titles. Every Olympic-style credential sits below a working-researcher baseline, because no named artifact ships with the medal, yet the ranking inverts at the marquee tier, where Nobel, Turing, and Fields laureates saturate the panel. For independent creators the tool out-ranks its maker, and the credential that does propagate is a named method or awarded paper. Being one of many named contributors to a celebrated artifact, by contrast, earns almost nothing -- the authors listed on a flagship model report or system card sit near the recognition floor -- because recognition attaches to the artifact's own distinctive name, not to the roster behind it. No bibliometric predicts recognition well; top-density institutions out-recognize peers at matched citations; and on 258 news events recognition loads on peak salience, not persistence. A self-report probe shows introspection reads a corpus prior, not its own knowledge.

Bojie Li, Noah Shi · 0 citations