Entropy-Preserving Linear Unit (EPLU): A Robust Activation Function for Stable and Efficient Training of Deep Neural Networks
TL;DR
Experimental results demonstrate that EPLU consistently outperforms state-of-the-art activation functions, including ReLU, Swish, and GELU, achieving 1~2% improvements in classification accuracy, and detailed analysis of training dynamics reveals that EPLU maintains more stable gradient norms, leading to improved convergence behavior and enhanced training robustness.
Abstract
Activation functions or the nonlinearity play a fundamental role in deep neural networks (NN) by governing gradient propagation, preventing neuron saturation and collapse, and preserving information flow across network layers. Since small changes in activation can greatly affect neural network performance, it is important to explore new activation functions and evaluate their potential impact. While widely used functions such as ReLU, Swish, and GELU have achieved remarkable empirical success, they often suffer from limitations including gradient instability, dead neurons, and suboptimal performance when applied to modern architectures such as vision transformers. Furthermore, the rapid evolution of deep learning models necessitates the development of adaptive activation functions that generalize effectively across diverse architectures. To address these challenges, we propose a novel activation function termed the Entropy-Preserving Linear Unit (EPLU), designed to maintain representational entropy, improve gradient flow, and enhance training robustness in deep networks. EPLU introduces a smooth, adaptive nonlinearity that mitigates saturation effects while preserving critical information throughout the network hierarchy. We evaluated EPLU within transformer-based architectures across multiple object recognition benchmarks, including CIFAR-10, CIFAR-100, and ImageNet-200. Experimental results demonstrate that EPLU consistently outperforms state-of-the-art activation functions, including ReLU, Swish, and GELU, achieving 1~2% improvements in classification accuracy. Furthermore, detailed analysis of training dynamics reveals that EPLU maintains more stable gradient norms, leading to improved convergence behavior and enhanced training robustness. These results highlight EPLU as a promising activation function for next-generation deep learning models, particularly in transformer-based architectures, offering superior performance, stability, and generalization.