Aug 2026· Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2· pp. 4441-4450· 0 citations· 8 references
TL;DR
GBDT-Guided Piecewise-Linear (GGPL) embeddings are introduced, which leverage GBDTs to provide a strong data-driven prior for breakpoint placement and fine-tune these breakpoints via gradient descent, which establishes GGPL as an effective numerical embedding for deep tabular regression.
Abstract
Tabular prediction is central to scientific and industrial decision-making, and many high-impact use cases are regression tasks that predict continuous targets. Deep tabular networks for regression tasks rely on effectively modeling irregular and non-smooth relationships between numerical features and continuous targets. To this end, numerous numerical embeddings have been proposed. However, their performance is often highly sensitive to parameter choices, such as the choice of frequencies for Fourier features and the placement of breakpoints for piecewise-linear embeddings. Moreover, the irregular and non-smooth characteristics of tabular data make it difficult for gradient-based end-to-end training to discover effective parameter settings for these embeddings. In this work, we demonstrate the underexplored potential of piecewise-linear embeddings, showing that finding better breakpoints alone can yield substantial gains. We introduce GBDT-Guided Piecewise-Linear (GGPL) embeddings, which leverage GBDTs to provide a strong data-driven prior for breakpoint placement. To further fine-tune these breakpoints via gradient descent, GGPL reparameterizes them for numerical stability and regularizes training via stochastic breakpoint deactivation. Across 28 regression datasets, integrating GGPL with diverse state-of-the-art deep tabular models yields consistent and significant improvements. GGPL is applied only at training time, introducing no inference overhead. Furthermore, in controlled MLP experiments, it uses only 0.4× as many breakpoints on average as prior piecewise-linear embeddings, while achieving higher accuracy. Combined with its negligible overhead, these results establish GGPL as an effective numerical embedding for deep tabular regression.
TabNSM provides an effective and scalable approach to deep tabular regression, and demonstrates that selective interaction modeling, structured regression supervision, and difficulty-aware sampling provide an effective and scalable approach to deep tabular regression.
Results support IAIML as a compact, interaction-aware framework appropriate for settings where bounded explanation size and controlled treatment of feature interactions are design requirements.
In this work, we introduce CoCo, a loss function aimed at learning normalized and well-structured representations. The proposed loss encourages intra-class collapse and inter-class contrast while preserving sufficient flexibility for neural networks to approximate geometrically optimal embeddings with large angular separation between classes. We provide a theoretical analysis positioning CoCo with respect to related objectives such as dot regression and cross-entropy, showing that the new proposed loss benefits from closer initialization to the optimal configuration, more informative gradients, and stronger incentives for class-wise representation collapse. Extensive experiments on diverse tabular datasets from the OpenML-CC18 benchmark show that CoCo achieves competitive performance with state-of-the-art methods, including kernel SVM, Random Forest, dot regression, and cross-entropy-based neural networks. In addition, both theoretical arguments and empirical analyses demonstrate that the proposal promotes tighter class clustering and faster convergence. These results highlight CoCo loss as an effective objective for learning discriminative representations while maintaining competitive predictive performance.
Blanca Cano-Camarero, 'Angela Fern'andez-Pascual, José R. Dorronsoro· 0 citations
A transfer parameterization is developed and validated for GNNs trained with SGD, Adam, and AdamW that yields stable feature updates, learning rate transfer, and improved performance as width and depth increase, and provides a practical recipe for scaling GNNs across a variety of learning tasks and training scenarios.
In deep learning for tabular data, efficient ensembles of multilayer perceptrons (MLPs) have recently emerged as effective and practical architectures. Existing methods of this kind use the same hyperparameters for all underlying MLPs, which requires hyperparameter tuning for achieving the best performance. In this work, we introduce TabPack, an efficient MLP ensemble with strong out-of-the-box performance and reduced reliance on traditional tuning. In a single run, TabPack samples and trains many MLPs with different hyperparameters efficiently in parallel and selects ensemble members on the fly during training. Thus, TabPack only requires specifying ranges from which to sample MLP hyperparameter rather than exact hyperparameter values, which naturally demands less precision for good performance. In experiments on medium-to-large public datasets, TabPack with default settings performs on par with extensively tuned prior methods, thus substantially reducing effort and compute resources needed to achieve competitive results on tabular tasks. Notably, running the default TabPack configuration on a modern MacBook took less time than tuning some baselines on an industry-grade GPU.
Yury Gorishniy, Akim Kotelnikov, Ivan Rubachev et al.· 0 citations
Understanding which parameters are influential in Large Language Models (LLMs) is central to improving their efficiency, reliability, and interpretability. We introduce Weight-Adjusted Gradients (WAG), a simple yet effective approach for estimating parameter importance that explicitly captures the interaction between model weights and first-order gradient information and identifies parameters that disproportionately influence model behavior, such as those responsible for collapse phenomena in LLMs. Across a range of models and settings, we show that WAG surfaces a tiny but critical subset of parameters whose modification leads to dramatic degradation in performance, a failure mode that existing importance metrics overlook. These findings reveal a previously underexplored interplay between weights and gradients, suggesting that parameter importance cannot be fully understood through either signal alone. The surprising effectiveness of WAG points to fundamental structural properties of trained networks and motivates new open questions about the role of zeroth-order and first-order information in deep learning. We demonstrate the practical utility of WAG across multiple applications, including expert allocation in mixture-of-expert architectures, parameter-specific unlearning, mixed-precision quantization, and layer selection for knowledge editing. Our results position WAG as a unified approach for analyzing, debugging, and controlling LLMs, and opens new directions for principled model-level interpretation.
