Cross-Domain Input, Mutual Exclusivity, and Inferential Reasoning: When LLMs Learn Words Like Humans
TL;DR
Structured cross-domain input enables LLMs to exhibit behaviour consistent with conditional IF–THEN rules, allowing them to infer and acquire novel words based on current visual contexts, suggesting that, with this input that integrates context and text, LLMs display adaptive, real-time human-like inferential reasoning in word learning.
Abstract
Humans acquire meaningful language by storing perceptual categories, category-word mappings, and conditional IF–THEN rules in rich, cross-domain, multimodal contexts. Crucially, structured cross-domain input that pairs visual context and language (text) appears to be fundamental to this process, enabling individuals to acquire, for example, the lexicon. The present study investigates whether Large Language Models (LLMs) can learn words when trained on structured cross-domain input rather than text-only exposure. This paper incorporates a controlled Fictitious-Animal Paradigm featuring 32 creatures and 32 pseudowords, divided into two phases: single-animal and dual-animal scenes. Single-animal contexts entailed storing categories and mappings, while dual-animal environments involved storing logical constraints and performing inference. The authors propose a computational model centred on four capacities for context-based word learning: storing perceptual categories, category-word mappings, and the Mutual Exclusivity rule (formalised as a conditional IF-THEN statement), and retrieving stored information to assign novel labels through inference. The evaluation demonstrates that structured cross-domain input enables LLMs to exhibit behaviour consistent with conditional IF–THEN rules, allowing them to infer and acquire novel words based on current visual contexts. These findings suggest that, with this input that integrates context and text, LLMs display adaptive, real-time human-like inferential reasoning in word learning.