The results support the common principle that explicit regulation of neuron utilization and effective connectivity can improve the emergence of generalizable internal computation and motivate broader investigation of biologically inspired activity regulation and adaptive sparsification, including in large language models.
Abstract
Grokking is a delayed transition from memorization to generalization that is often accompanied by substantial reorganization of internal representations. This paper studies whether biologically inspired mechanisms, many of which are not commonly incorporated into artificial neural networks, can actively promote this transition by regulating hidden-layer computation at the levels of neuronal activity, response, and effective connectivity. We augment a multilayer perceptron with input gating, structural plasticity, gain modulation, threshold modulation, homeostasis, lateral inhibition, and activation decorrelation, and evaluate these mechanisms through systematic ablations on two established grokking benchmarks: sparse parity and noisy XOR classification. The results show that the mechanisms contribute unequally to generalization. Homeostasis provides the strongest and most consistent benefit, while structural sparsification emerges as the second major mechanism. The remaining biologically inspired mechanisms have smaller or less consistent effects in the present experiments. For both problems, the results support the common principle that explicit regulation of neuron utilization and effective connectivity can improve the emergence of generalizable internal computation. These findings motivate broader investigation of biologically inspired activity regulation and adaptive sparsification, including in large language models, where they may accelerate the development of generalizable representations and reduce the optimization time required for robust generalization.
Biologically plausible learning models aim to explain how neural circuits can implement effective learning under the constraints of real neurons. Although significant progress has been made, a major remaining challenge is that existing models often allow neurons or synapses to represent mixed-sign values, both positive and negative, in violation of a basic aspect of cortical circuitry -- Dale's constraint: biological neurons are either excitatory or inhibitory, but not both, and synapses cannot change sign. In this work, we address this discrepancy by introducing a biologically motivated neural architecture in which both neural activations and learning signals are represented by non-negative activity, and synapses have fixed sign, while still supporting backpropagation-like learning. Our approach uses two complementary interacting non-negative channels to represent positive and negative contributions, inspired by evidence of on-off representations in the brain. These channels are implemented through a simple neural circuit motif, which is repeated throughout the network in both bottom-up and top-down pathways. Combined with a local Hebbian learning rule, the resulting model propagates learning signals and updates weights using only local interactions between neurons. We show theoretically that our learning scheme can exactly recover the backpropagation update despite relying solely on non-negative error signals. Empirically, beyond satisfying stronger biological constraints, the on-off architecture learns efficient representations, yielding substantial gains over comparable vanilla networks on the Tiny ImageNet benchmark. These results demonstrate that effective learning can emerge from biologically plausible mechanisms without requiring mixed-sign signals, providing a step toward more realistic models of neural computation.
The brain can rapidly perform perception, prediction, and decision-making tasks. The capabilities stem from the collective coordination of neurons. This collective activity forms a manifold structure that naturally supports information representation and prediction, yet most neuromorphic work ignores this structure. Here, we construct an artificial neural manifold based on Mott memristors, enabling accurate and robust prediction while reducing the number of required samples. By constructing an artificial neuron circuit, a bell-shaped tuning curve similar to that of biological neurons is obtained. The tuning curve converges the large-scale neuronal firing into a compact, low-dimensional manifold structure. This structure satisfies the delay embedding theorem to establish a spatiotemporal information (STI) equation, enabling rapid prediction of neural activity with small sample sizes. In addition, we introduce a memory factor to modify the STI equation, which improves prediction accuracy and robustness. We not only accurately perceive incomplete images but also predict epileptic seizures. Inspired by collective neuronal activity, Wang et al. develop a Mott-memristor-based hardware system that maps complex spike signals to population-level artificial neural dynamics, enabling accurate prediction from limited data.
Rui Wang, Guolei Liu, Saisai Wang et al.· Nature Communications· 0 citations
Biological neural systems achieve high efficiency and robustness through compartmentalized architectures. In contrast, modern artificial neural networks rely on globally entangled structures, which obscure decision logic and suffer from catastrophic forgetting. Here, we report a Decomposable Spiking Neural Network (D-SNN) that eliminates global synaptic entanglement by structurally isolating classification pathways into independent experts. Optimized via a bio-inspired push-pull loss function, the D-SNN achieves competitive accuracies on MNIST, Fashion-MNIST, and CIFAR-10/100 benchmarks. This modular approach matches the performance of fully dense networks while utilizing an order of magnitude fewer parameters. In addition, our networks operate with up to several orders of magnitude lower firing rates and fewer synaptic operations. Furthermore, physically severing connections between experts provides inherent protection against catastrophic forgetting during sequential learning. Crucially, these isolated pathways generate auditable neural signals, increasing decision transparency. This biomimetic, verifiable architecture establishes an efficient foundation for deploying deterministic neuromorphic intelligence in resource-constrained edge environments.
Maksim Bazhenov, S. Grubas, V. Putkaradze· 0 citations
Self-organization is an emergent property of life, driven by the collective behavior of individual components acting on local information. Biological neurons, through local interactions transmitted through synapses, are able to learn efficiently and can adapt their connections over an organism's lifespan. Motivated by these desirable properties of adaptability and local interaction, neural cellular automata (NCA) models have been successful at learning morphogenesis solely through local update rules, demonstrating stability over many updates and robustness to perturbations. In this work, we introduce Meta Neural Cellular Automata (MetaNCA), a framework that learns local rules which self-organize the weights of artificial neural networks. A learned rule network iteratively updates the weights of a task network using only local interactions on the computation graph. We propose a novel Weight Transformer architecture for the local rule network, which uses linear attention to aggregate signals from neighboring weights and hidden states. Once trained, the rule network generates task networks of diverse architectures without backpropagation. We show that MetaNCA generates weights for feedforward MLPs, CNNs, and ResNets on MNIST and CIFAR-100, scaling to networks of 2 million parameters. We further show that MetaNCA generalizes to architectures not seen during meta-training, and that architectural diversity in the training phase strengthens this generalization.
Meet Barot, Daniel Berenberg, Sina Khajehabdollahi· 0 citations
Grokking is a phenomenon in which neural networks initially memorize training data and only later exhibit strong generalization after prolonged optimization. Despite extensive recent study, the factors influencing the emergence and timing of grokking remain incompletely understood. We investigate the relationship between representation geometry and delayed generalization. We find that dimensionality collapse consistently precedes the onset of grokking in all evaluated settings. Motivated by these observations, we introduce Geometric Dimensionality Regularization (GeomDR), a simple spectral regularizer that modifies the effective dimensionality of hidden representations during training. Across modular addition, modular division, and permutation composition tasks, GeomDR consistently alters grokking dynamics and can substantially accelerate the onset of generalization depending on the intervention schedule and target dimensionality. In several settings, grokking is accelerated by up to 52 times relative to standard AdamW training. Similar qualitative effects are observed in both multilayer perceptrons and transformers. Together, these results suggest that representation geometry can serve as an effective control signal for grokking and provide evidence that geometric interventions offer a practical approach for studying and influencing delayed generalization in neural networks.
Most neurons release either excitatory or inhibitory neurotransmitters. However, multiple inputs to the lateral habenula (LHb) co-transmit glutamate and GABA, transmitters with opposing effects on LHb output. Although the LHb has an established role in reinforcement learning, the adaptive significance of glutamate/GABA co-release remains unclear. Using biophysically realistic simulations, we show that GABA co-release is sufficient to produce temporal difference (TD)-like transformations of input activity, computations commonly used for reinforcement learning and behavioral optimization. Heterogeneous GABA-to-glutamate ratios, like those found among LHb neurons ex vivo, produce diverse TD-like computations linked to higher-order decision-making. Single-cell RNA-sequencing analysis and machine-learning image analysis further indicate that glutamate/GABA co-release expanded across vertebrate evolution, from fish to mice, rats, and monkeys. Evolutionary expansion of glutamate/GABA co-release may have supported increasingly sophisticated learning and decision-making that contribute to intelligent behavior.
Natalia Rodríguez-Sosa, Lupita Rios, You-Hsin Lin et al.· bioRxiv· 0 citations