A Cognitive Framework for Idea Diversity in Large Language Models

Abstract

Consumers, marketers, and new product developers increasingly rely on LLMs for ideation. Yet ideas generated by independent LLM sessions tend to be less diverse than ideas generated by independent samples of humans, raising concerns that widespread reliance on LLMs could homogenize marketplace ideation and decrease consumer welfare. We identify two mechanisms undermining LLM idea diversity, informed by theories of creative ideation from new product development and cognitive psychology. First, LLMs aggregate knowledge into a unified distribution rather than exhibiting the knowledge partitioning inherent to human populations, where each person occupies a distinct region of the knowledge space. Second, at the individual level, LLMs exhibit fixation just as humans do, where early outputs constrain subsequent ideation. Two product-ideation experiments demonstrate that targeted prompting interventions can address both mechanisms. Persona modifiers based on ordinary individuals improve knowledge partitioning but amplify fixation. This fixation can be mitigated by using chain-of-thought (CoT) prompting, and combining both strategies produces the highest idea diversity, outperforming humans. These findings offer a theoretically grounded framework for understanding LLM idea diversity and practical strategies for leveraging AI's efficiency without compromising the idea diversity that sustains marketplace variety and consumer welfare.