HY-CNN-GRU: A Universal Deep Learning Framework for Robust Channel Estimation Across Heterogeneous Wireless Environments in 6G NOMA Systems

Abstract

This paper proposes a novel hybrid Convolutional Neural Network-Gated Recurrent Unit (CNN-GRU) architecture for accurate channel state estimation in 2-user Non-Orthogonal Multiple Access (NOMA) systems operating under diverse wireless propagation environments. The proposed model integrates convolutional layers for effective spatial feature extraction from received pilot-assisted signals with gated recurrent units to capture temporal dependencies, enabling robust offline training and online joint symbol detection. The continuous channel state information (CSI) is subsequently recovered via a Least Squares (LS) back-calculation step, ensuring accurate continuous channel estimation. Comprehensive evaluations are conducted across four representative channel models—Multipath (frequency-selective fading), Rician (line-of-sight dominant), Nakagami-m (variable fading severity), and millimeter-wave (mmWave, high path loss and blockage-prone)—under varying cyclic prefix (CP) lengths and pilot densities. Simulation results demonstrate that the hybrid CNN-GRU significantly outperforms conventional Least Squares (LS) and Minimum Mean Square Error (MMSE) estimators in terms of Mean Square Error (MSE), Symbol Error Rate (SER), and derived classification metrics (accuracy, precision, recall, and F1 score). Substantial performance gains are achieved, particularly with increased pilot density (up to 64 pilots) and extended CP (20), yielding MSE reductions of 40–84% and near-perfect classification accuracy (>0.99 at SNR=20 dB in optimal configurations). The proposed approach exhibits remarkable robustness across all channels, with the most pronounced improvements in challenging Nakagami-m and mmWave environments, where conventional methods struggle due to severe fading and propagation impairments. These findings highlight the efficacy of the hybrid deep learning framework in enhancing channel estimation reliability for practical NOMA deployments in next-generation wireless systems.