Jürgen Teich

Rethinking Neural Nonlinearity as Gating

This paper demonstrates that standard activations -- whether piecewise-linear (ReLU, PReLU, Hardtanh) or smooth (SiLU, Sigmoid, Tanh, GELU) -- are in fact instances of a single Threshold Gating (TG) primitive, and proposes a'Minimal Branch Theorem' which relates the minimum number of required branches in the authors' primitive to the trainability of general deep neural networks.