Skip to content
Preprint

Are LLMs becoming similarly creative? Evidence from three years of models

Aug 2026 · 3 citations · 39 references
Computer Science

TL;DR

A preliminary analysis of LLM creative outputs spanning three years of model releases shows a statistically significant decrease in model output diversity over time, suggesting that LLM outputs may be converging in creative substance across models.

Abstract

Many benchmarks track Large Language Model (LLM) performance on tasks with verifiable answers, but less is known about how LLM performance is evolving on open-ended tasks, where creativity, originality and diversity may matter as much as quality. As LLMs increasingly support human ideation and creative work, understanding trends in LLM performance on open-ended tasks is critical. This paper presents a preliminary analysis of LLM creative outputs spanning three years of model releases, examining model responses to Infinity-Chat100, a real-world collection of open-ended user queries, and the Alternate Uses Task, an established psychometric creativity assessment. Using sentence-embedding similarity, we examine trends in LLM responses to these prompts. Our findings show a statistically significant decrease in model output diversity over time, suggesting that LLM outputs may be converging in creative substance across models. If this trend persists, LLM-driven homogenization may progressively diminish human agency in human-AI co-creative work, demanding careful consideration of LLMs'role in the human creative process.

View source

Similar papers

Review Open access Sep 2026

Evaluation of Large Language Models as Tools, Models, and Partners in Creative Thinking Research: A Selective Narrative Review with the GCA Framework

Creativity research faces three persistent bottlenecks: divergent-thinking scoring is labour-intensive, cognitive models of creativity remain underspecified, and laboratory tasks fall short of real-world creative achievement. Large language models (LLMs) offer potential solutions, but the field lacks a structured frame...

Ke-Xin Huang, Chun-Lei Liu, Jia-Qin Yang · 0 citations
#natural language process... Preprint Sep 2026

The Effects of Incremental Instruction Delivery on Language-Model Creative Writing

Large language models are increasingly used as interactive writing tools, where users develop stories, revise ideas, and introduce new requirements across multiple turns rather than specifying a complete brief upfront. Yet most evidence on multi-turn instruction degradation comes from tasks with objectively verifiable...

Anshuman Singh, Abrar Eyasir, Haseeb Yaqoob et al. · 0 citations

What Do We Expect from LLMs? Mapping the Design of LLM Benchmarks

This work systematically map 14,767 papers introducing or updating evaluation resources from arXiv submissions between January 2022 and August 2026, using staged screening and automated full-text coding to examine changes in target systems and domains, evaluation materials and conditions, and scoring mechanisms.

Chao Wang · 0 citations
#small language model Open access Sep 2026

Human versus machine

Although LLM-based benchmark ratings can approximate expert judgments and reduce the need for labor-intensive human triple coding, limitations remain regarding cost, genre- and task specificity, and sensitivity to text presentation and student grade level—factors that constrain immediate classroom use, particularly for...

Afra Sturm, Valentin Unger, Fabian Grünig · 0 citations
Preprint Aug 2026

The Limits of Automatic Evaluation of Creativity in Large Language Models

Large Language Models (LLMs) are increasingly capable of generating text that challenges human performance in domains requiring creativity, yet evaluating creativity in LLM-generated content remains a significant challenge. Here, we investigate whether current automatic evaluation methods can reliably capture human jud...

Alessandro Tutone, Giorgio Franceschelli, Mirco Musolesi · 1 citation

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.