Low-Carbon Urban Freight Optimization: Per-Order Adaptive Mode Mixing with Demonstration-Regularized Constrained Reinforcement Learning
Urban last-mile delivery is a rapidly growing source of city-centre emissions, and decarbonizing it without eroding service quality has become imperative for climate goals. Operators are turning to multimodal systems that integrate road vehicles, off-peak metro freight, and electric drones—yet the optimal delivery channel varies dynamically with location and time. Current RL-based schedulers handle constraints via manually tuned penalty weights, lacking formal safety guarantees, and the feasibility of online carbon-cap enforcement under partial observability remains an open question. To address this, we model the problem as a Constrained Markov Decision Process (CMDP) and propose a demonstration-regularized Lagrangian deep RL algorithm. Our approach learns an online policy that is model-free at deployment—it controls emissions in expectation against a hard carbon budget, makes per-order decisions using only state observations, and operates without an emission model at test time (the demonstrator used at training time does access the emissions model, so “model-free” refers strictly to the deployment phase). Experiments on synthetic benchmarks and a Nanjing-inspired scenario—grounded in real metro topology and population-weighted demand—show that our policy achieves emissions within 1.3% of the offline optimum. It robustly tracks a ±17% carbon-budget band across a threefold daily volume range and a threefold city-scale range, with zero per-instance tuning. By contrast, a standard PPO with fixed penalty weights consistently degrades to single-mode selection. Our findings suggest that hard carbon budgets can be controlled in expectation online at modest cost—a step toward operator-facing low-carbon logistics whose average emissions honour a binding carbon budget, though external validation on operational data and a risk-sensitive formulation that upgrades this average control into per-day compliance are still required before deployment.