Romit Chatterjee

The Information Shadow: Measuring Structural Limits on What Language Models Can Learn

The information shadow is introduced: the region of phenomena that a text-trained learner cannot acquire regardless of scale, comprising (I) structures language cannot express, (II) functions that are statistically non-identifiable from the training distribution, and (III) functions that are representable but unreachable by gradient-based training.