Entropies and Negentropies from f -Divergences and Their Application to Dimensionality Reduction

Abstract

Distributions are ubiquitous across scientific disciplines, extending well beyond probability and statistics. In machine learning, finite probability distributions arise naturally as the softmax output layers of convolutional neural networks and large language models, where they encode class probabilities in image classification and token probabilities in language generation. When such distributions are high dimensional, however, storage and computational costs become significant. In previous work, we introduced two families of generalized entropies derived from f -divergences, using majorization as a reference framework for comparing distributional homogeneity. In this paper, we extend that framework in several directions. First, we study majorization relationships between subcompositions of a distribution. Second, we introduce generalized negentropies derived from f -divergences and analyze their role alongside entropies in dimensionality reduction. Third, we embed both entropies and negentropies into the setting of Shannon’s information channel and show that the classical channel identities are satisfied exclusively by Shannon entropy. These results provide a unified information-theoretic framework for dimensionality reduction of finite distributions, clarifying the structural role of f -divergence-based entropies and negentropies and their relation to classical information measures.