The entropy space theory offers a novel priori framework for mathematical fundamentals of deep learning and a unified coordinate system is proposed based on the theory.
Abstract
Deep learning is often criticized for its theoretical research lagging behind practice. To make deep learning easier to understand, the entropy space theory is first introduced here. The entropy space can cover all the possibilities of any deep learning model by topological structure. It is independent of network parameters. Through the designed fundamental operations and norm, entropy space is proven to be a normed space within the formal axiomatic framework. Based on the theory, a unified coordinate system is proposed. It can coordinatize every state of a model and rank them by compression of the maximal value of information entropy. The theory offers a novel priori framework for mathematical fundamentals of deep learning.
Experimental results demonstrate that the proposed TN-based universal entropy coding method consistently outperforms existing universal approaches and is comparable to task-specific approaches in the tasks of lossless graph and image compression and learned lossy image compression.
Xiao-Xuan Fan, Wen-Rui Dai, Shaohui Li et al.· IEEE Transactions on Signal...· 0 citations
The study demonstrates that the VAE possesses significant advantages in state compression and uncertainty modeling, but suffers from issues such as blurry image generation and oversimplified posterior distribution assumptions.
Xiao-Zhou Gao· Theoretical and Natural Scie...· 0 citations
This survey takes a complementary, theory-grounded perspective, examining interpretability through four classical machine learning (ML) frameworks, each grounded in a mature mathematical theory, and concludes with open problems and directions for scalable, theory-grounded interpretability.
M. U. Khalid, Abeer A. K. Alharbi, Shariq Bashir et al.· Journal of King Saud Univers...· 0 citations
Stochastic neural network broadens deep learning from prediction‐oriented function approximation to a richer framework for statistical inference, and two related architectures are discussed: kernel‐expanded StoNet and sublinearly structured DNNs, which can be viewed as structural or theoretical extensions of the StoNet...
A broad class of neural networks trained under the $\ell^2$ loss by gradient descent with weight decay with weight decay is studied, and the convergence of GD to a neighbourhood of the global minimizers of the empirical loss is proved.
Yu-Qing Wang, I. Kevrekidis, Mikhail Belkin· 1 citation
We investigate a general-purpose layer for deep learning that, instead of compressing arbitrary-size training data into fixed-size weight matrices, stores a new pair of key-value representations for every data point during training, and retrieves and recombines these representations through an attention mechanism at in...
Maximilian Schlegel, Rajai Nasser, Seijin Kobayashi et al.· 0 citations
