Skip to content
Preprint

Quantifying Depth Sufficiency in Residual Neural Networks: A First-Order Criterion

Aug 2026 · 0 citations · 33 references
Computer Science

TL;DR

This work defines first-order residual depth saturation as the absence of a strict local decrease from every admissible insertion, and proves residual non-degeneracy is necessary and sufficient: additional depth has first-order value exactly when conditional activation gradients have a nonzero projection onto at least one admissible residual tangent space.

Abstract

How can we determine whether a trained neural network is already deep enough? We study this under a fixed function-preserving residual-growth protocol specifying insertion locations, residual families, zero-output initializations, and zero-state first-order updates. We define first-order residual depth saturation as the absence of a strict local decrease from every admissible insertion. We prove residual non-degeneracy is necessary and sufficient: additional depth has first-order value exactly when conditional activation gradients have a nonzero projection onto at least one admissible residual tangent space. This boundary is shared by descent-compatible zero-state updates and invariant under regular local reparameterizations preserving that tangent space. Under residual-signal realizability, raw activation-gradient vanishing exactly certifies saturation. Across ResNets, GPT-2-style models, and continued-pretrained Pythia checkpoints, the maximum activation-gradient norm decreases toward a low-signal regime with depth. Function-preserving growth also achieves converged performance competitive with training from scratch. These results support activation-gradient magnitude as a conservative diagnostic of the remaining empirical first-order value of residual depth.

View source

Similar papers

#machine learning Preprint Sep 2026

Minimal-Norm Univariate Two-Layer ReLU Classification: Exact Solutions and Global Optimality with Skip Connections

It is shown that adding a free affine skip connection leaves these function-space solutions unchanged but fundamentally improves the parameter-space landscape: every KKT point of the constrained problem becomes globally optimal, whereas suboptimal KKT points can occur without the skip connection.

Karolina Drabik, Ben Lewis, Antoni Puch et al. · 0 citations
#machine learning Preprint Sep 2026

Reducing the Adaptation Gap Through Reachable Fisher Geometry

Parameter-efficient fine-tuning (PEFT) determines not only how many parameters are trained, but also which local directions a model can move in, so similar adapters can affect subgroup losses differently. Since curvature matrices are infeasible to form at adapter scale, scalar summaries such as the Fisher trace are oft...

Wasif Jalal, Sachin Deb, Asif Salekin · 0 citations
Preprint Aug 2026

Iterative Erasure Count Is Not an Affine-Invariant Concept Dimension

How many directions does a neural representation use to encode a concept? A common answer repeatedly erases probe directions and reports the stopping count or cumulative removed rank. We show that both quantities can change under an information-preserving invertible reparameterization, so neither is intrinsically a con...

Tingan Jin, Shu-Hang Dong, Hao-Song Li et al. · 1 citation
#artificial intelligence Preprint Oct 2026

What to Preserve in Recursive Computation: A Local Predictive Sufficiency Principle

Recursive computation repeatedly compresses or reuses intermediate states, creating a simple tension: information that must remain useful across longer recursive paths is also exposed to more opportunities for loss before reaching the final prediction. Existing reconstruction or local-prediction objectives provide trac...

Pei-Lin Wang, Shi-Yang Feng, Hong-Fu Gao et al. · 0 citations
#machine learning Preprint Sep 2026

RINS: Residual-Image Neural Subspace Solvers for Large Sparse Linear Systems

Large sparse linear systems from PDE discretizations require correction subspaces whose operator images explain the current residual. We study this residual-image viewpoint and propose Gate-RINS, a neural subspace solver that generates polynomial correction bases from cached residual probes and modulates them with a li...

Zhong-Yan Ouyang, Wei-Xin Liao, Ming-Quan Feng et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.