Megha Thukral

Attention Function as an Intrinsic Inductive Bias: How Models'Behavior Diverges in Novel Contexts

Developmental psychology holds that certain priors are given to infants prior to experience rather than induced from data, and that the influence of such priors is suppressed under strong, well-constrained conditions but reasserts itself under weak ones. We ask whether an analogous principle holds for the Transformer:...