Sparse Competition during Training For the Emergence of Specialized Modules

Weight initialization shapes task organization in recurrent neural networks

It is found that networks with low initial weight variance converge to solutions in which different subtasks rely on largely overlapping weight subspaces, whereas high-variance networks implement subtasks in higher-dimensional, more segregated weight subspaces.

The Ups and Downs of Backprop Weights

It is argued that functional parameter identifiability may provide an organizing principle for models that systematically reuse and recombine learned functions while adapting only the components that need to change.

CORTEX: Learning to Share and Specialize in Dense Language Models

Large language models are trained on heterogeneous data mixtures, where different knowledge domains require both shared knowledge and specialization. Existing modular approaches typically impose explicit components or discover modules through interpretability analysis after training. In this work, we propose CORTEX, a...

Expert Systems With Applications

Classifications in modular restricted Boltzmann machines

We consider a modular associative neural network made of $L$ Hopfield models (HMs), coupled so that intra-module interactions are Hebbian and inter-module interactions are anti-Hebbian; this competitive coupling is known to endow the network with pattern-disentanglement capabilities. The integral representation of this...

Topographic Training Concentrates Causal Circuits Without Improving Neuron Monosemanticity

Mechanistic interpretability of vision transformers seeks to decompose model computation into human-readable units, but learned representations entangle many concepts in each neuron. Feature superposition is widely treated as the central obstacle to this decomposition, yet most mitigations (sparse autoencoders, diction...