Development of a Hierarchical Reinforcement Learning Model for Dynamic Retail Pricing and Cooling Demand Response in Building Communities
TL;DR
A two-level hierarchical Soft Actor–Critic framework to quantify how a virtual retail aggregator and a centralized building controller can coordinate bounded internal retail prices, shared battery scheduling, and cooling demand response supports an economic and emissions tradeoff in the stated six-building simulation.
Abstract
Building cooling loads provide flexibility for demand response, but their operation must balance user cost, aggregator revenue, carbon emissions, and thermal comfort. Existing building control studies often treat electricity prices as external inputs, whereas many dynamic pricing studies simplify building thermal responses and comfort constraints. This paper develops a two-level hierarchical Soft Actor–Critic framework to quantify how a virtual retail aggregator and a centralized building controller can coordinate bounded internal retail prices, shared battery scheduling, and cooling demand response. The upper controller acts first, and the lower controller then jointly adjusts the cooling actions of six buildings in response to the retail price and comfort states. In the original 92-day full-period comparison, the selected multi-agent reinforcement learning (MARL) policy reduces user cost by 14.1%, transfer-neutral external settlement cost by 31.3%, average temperature error by 29.0%, and carbon emissions by 27.4% relative to the default static retail baseline, while maintaining positive aggregator profit. A separate chronological robustness protocol uses 60 days for training, 16 for validation-based checkpoint selection, and 16 completely unseen days for testing. Across three random seeds on the unseen test period, MARL increases aggregator profit by 24.1% and reduces user cost, transfer-neutral net cost, and carbon emissions by 4.0%, 24.5%, and 17.4%, respectively, although mean temperature error increases by 10.9%. On the unseen test, the price-responsive static rule achieves lower user cost, net cost, temperature error, and carbon emissions than MARL, whereas MARL achieves higher aggregator profit. The results therefore support an economic and emissions tradeoff in the stated six-building simulation rather than universal dominance across all metrics or building communities.