Heuristic-Supervised-DRL: A Unified Optimization Framework With Convergence Analysis

Abstract

Many real-world problems involve hierarchical multi-objective optimization over coupled sequential decisions. A common strategy is to combine heuristic planning with deep reinforcement learning (DRL). While existing hybrid methods often show strong empirical performance, the coupled closed-loop learning dynamics between upper-tier planning and lower-tier control are typically analyzed only empirically, and explicit convergence guarantees for the integrated scheme remain limited. To address this gap, we propose a heuristic-supervised-DRL (HSD) framework that tightly couples (i) a heuristic planner for upper-tier decision-making, (ii) a DRL agent for lower-tier execution, and (iii) an online supervised predictor that serves as an adaptive bridge between planning and execution. The key novelty of HSD lies in this closed-loop architecture and its accompanying theoretical treatment. By formulating the coupled updates as a two-timescale stochastic approximation process, we show that, under standard conditions, the supervised predictor tracks a quasi-stationary regression target and the overall joint process converges almost surely to an asymptotically stable equilibrium. We further analyze robustness under approximate planning errors. As a case study, we instantiate HSD in a multi-UAV-assisted mobile edge computing system. Experimental results show that the proposed framework consistently outperforms representative baselines in key performance metrics, demonstrating both its practical effectiveness and its value as a principled framework for hierarchical optimization in dynamic environments.