A Deep Latent Variable Framework for Jointly Modeling Missingness, Measurement Error, and Heterogeneity
TL;DR
A unified probabilistic framework that jointly addresses missing data, measurement error, and population heterogeneity utilizing deep latent variable representation is proposed that integrates a novel hierarchical tree-routed variational autoencoder with pattern-aware latent representations and calibration-based denoising.
Abstract
Missing data, measurement error, and population heterogeneity are pervasive challenges in analyzing data arising from modern observational studies and machine learning applications. Although these problems frequently coexist and interact, they are often treated separately in existing works. We propose a unified probabilistic framework that jointly addresses these issues utilizing deep latent variable representation. The proposed method integrates a novel hierarchical tree-routed variational autoencoder with pattern-aware latent representations and calibration-based denoising. The framework accommodates missing data mechanisms, including MCAR, MAR, and MNAR, while simultaneously learning subgroup-specific and globally shared latent structure. The introduced reconvergent routing mechanism enables selective parameters to be shared across related subpopulations, which offers flexibility as well as improved statistical efficiency. Simulation studies demonstrate substantial improvements over existing deep generative imputation approaches under complex heterogeneous missingness and measurement-error settings. The proposed framework provides a principled approach for learning from noisy and incomplete data in modern healthcare and other high-dimensional applications.
