This study adapts an open-weight reasoning model through reinforcement learning with verifiable rewards (RLVR), which converts exact offline dynamic-programming action values into dense rewards for every candidate action to adapt open-weight reasoning models to building storage scheduling.
Abstract
Buildings are expected to shift cooling loads in response to grid conditions. Thermal energy storage (TES) enables this shift, but scheduling it well requires planning hours ahead under storage constraints. Model predictive control (MPC) and reinforcement learning are difficult to scale across buildings. This study instead adapts an open-weight reasoning model through reinforcement learning with verifiable rewards (RLVR). We convert exact offline dynamic-programming (DP) action values into dense rewards for every candidate action. Using only 30 training prompts, reinforcement fine-tuning (RFT) trains the model as an upper-level scheduler that outputs hourly heat-pump setpoints from text-based states and forecasts. Evaluation uses a deliberately simple office-building TES benchmark where exact DP is tractable and the optimum is known. RFT reduces the open-weight model's emissions from 70.5 to 61.2 kg-CO2, close to the DP optimum of 60.8 kg-CO2. GPT-5 nearly matches DP and MPC without task-specific training, while GPT-4o, a non-reasoning LLM, produces higher emissions than the no-storage baseline, so inference-time reasoning appears important. Trace analysis shows that RFT mainly stabilizes observable planning patterns (candidate comparison, look-ahead, and feasibility checking) rather than creating a new strategy. Robustness and generalization tests clarify what transfers: the reinforced planning patterns persist under forecast errors and an unseen TES condition and carry over to a battery task, but its different structure limits the gains. DP-based verifiable rewards offer a practical way to adapt open-weight reasoning models to building storage scheduling. These results motivate higher-fidelity tests of whole-building control and scalable verifiers for city-scale energy management.
Multi-zone variable-air-volume control must balance thermal comfort, indoor air quality, and electricity use across several continuous actuators. Model predictive control and reinforcement learning are widely studied, but deployment typically requires building-specific modeling or training, limiting scalability. We first test whether a frontier reasoning model (an LLM trained to use additional inference-time computation) can achieve competitive VAV control from text without building-specific training. With that capability established, we then test whether TD3-guided reinforcement fine-tuning (RFT) can transfer control knowledge into a locally deployable open-weight model. Five controllers are evaluated over three summer days in a physics-based four-zone emulator. Relative to a Guideline 36-based baseline, TD3 reduced HVAC electricity by 4.5% while improving temperature and CO$_2$ compliance. Without building-specific training, GPT-5 achieved the largest reduction (6.2%) but reduced the ventilation margin. For RFT, deterministic rollouts restore a saved state, apply one candidate, and follow TD3 to score each action. Auditing a learned critic against these rollouts exposed a failure hidden by its near-perfect across-time correlation ($r=0.9998$): within-state ranking was unreliable; the critic selected the rollout-best candidate in only 5 of 10 states. Even with the rollout verifier, 200 RFT steps produced no sustained improvement in sampled-action return; the open-weight controller used more electricity than the baseline before and after training, and its five-minute predictions remained worse than persistence. GPT-5 predicted transitions far better. Exact rollout scores rank sampled actions but reveal neither next-state effects nor an improvement direction. The unchanged transition errors motivate transition-focused supervised fine-tuning before value-based RFT.
Takumi Shioda, Kohei Terashima, T. Nagai· 0 citations
HVAC systems represent a major share of building energy consumption. Traditional control strategies are limited in coordinating energy-comfort tradeoffs across multiple zones simultaneously. Reinforcement learning (RL) offers adaptive, data-driven control that optimizes performance over time. However, deploying learned neural network controllers in safety-critical building systems remains challenging due to lack of formal safety guarantees. We propose a safety-certified deep RL framework for multi-zone residential HVAC control. Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC) agents are trained in an EnergyPlus/Sinergym simulation to minimize energy consumption while maintaining thermal comfort. Post-training safety certification is performed on the PPO policy using Lipschitz-based forward invariance analysis, building on existing tools for the computation of Lipschitz constants for neural networks, to guarantee constraint satisfaction. Both agents are evaluated over an annual simulation cycle in an eight-zone variable refrigerant flow (VRF) testbed. The PPO agent achieves 67\% comfort violation reduction compared to rule-based control, while the SAC agent achieves 27.6\% energy savings. The PPO policy satisfies formal safety certification with a margin of $2.003^\circ$C. These results demonstrate the feasibility of combining reinforcement learning with post-training safety verification for multi-zone building control.
Oussama Ziadi, A. Rochd, S. I. Kaitouni et al.· 0 citations
Deploying reinforcement learning (RL) agents for heating, ventilation, and air-conditioning (HVAC) control across climatically diverse buildings is challenging, as policies trained in one climate often fail to generalize to others. We investigate transfer and meta-reinforcement learning for building climate control using Sinergym with a custom EnergyPlus model of a real room, comparing Double Deep Q-Network (DDQN) and Model-Agnostic Meta Learning DDQN (MAML-DDQN) across three transfer scenarios on a target very hot climate. Fine-tuned DDQN reduces temperature violations by a factor of 35 over the from-scratch baseline and improves reward by 86% over zero-shot transfer, while MAML-DDQN converges in less than half the fine-tuning episodes and exhibits substantially lower variance across seeds. Results highlight a key trade-off: MAML-DDQN is preferable at adaptation-focused deployments, while fine-tuned DDQN is superior when maximum performance is the objective.
Ulaş Filiz, Sajjad Baghaee, Mubeen Ahmed Fayyaz et al.· Signal Processing and Commun...· 0 citations
The rapid growth of energy markets and demand-side response programs has created a significant need for intelligent building-level control strategies to capture high volatility energy consumption in response to price signals and grid conditions. This paper presents a reinforcement learning (RL)-based building energy management framework that models commercial buildings as active, grid-interactive assets capable of providing real-time flexibility while maintaining occupant comfort. The proposed approach integrates historical and real-time data from IoT sensors, HVAC systems, and weather forecasts to build an adaptive environment for RL agents. The RL model is applied to learn optimal control policies that minimize operational energy cost in response to flexibility markets through load shifting, peak shaving, and short-term demand response actions. The framework also incorporates a forecasting module to predict 30-minute interval energy consumption using deep learning, enabling proactive decision-making under uncertainty. Results from simulation experiments demonstrate that the RL agents achieve significant cost savings compared to rule-based control strategies and offer a reliable, automated control to unlock underlying flexibility within building systems. The paper discusses problem formulation, algorithmic development, simulation workflows, comparative metrics, and practical deployment considerations for Saudi Arabia's smart city initiatives.
Abdulaziz Almalaq· 2026 6th International Confe...· 0 citations
Advanced control of heating and cooling systems can substantially reduce energy costs and pollution. However, real-world adoption of popular algorithms among researchers, such as model predictive control (MPC) and reinforcement learning (RL), remains limited due in part to their high deployment and commissioning costs. Here, we develop two nearly commissioning-free controllers tailored to objectives that depend linearly on the controlled thermal load, such as energy costs and pollution. The controllers require at most two thermal parameters. In representative heating simulations, controller performance is robust to large parameter specification errors, suggesting potential for deployment with no tuning. The controllers maintain good occupant comfort while achieving 43 to 98% (depending on the electricity pricing and controller variant) of the performance improvement achieved by an omniscient policy with perfect model information and forecasts. These results suggest that simple, structure-exploiting controllers may capture most of the attainable value of advanced control while avoiding the data, modeling, tuning, and computational burdens that can arise with conventional MPC or RL.
W. G. Dierking, Arash Khabbazi, L. D. Premer et al.· 0 citations
Hybrid renewable power plants that combine multiple generation technologies with energy storage can improve renewable utilization and reliability, but their operation requires solving a high-dimensional, sequential dispatch problem under uncertainty. This paper presents a reinforcement learning (RL) framework for operational dispatch of a hybrid concentrated solar power (CSP)–wind system–Photo Voltaic (PV) solar system equipped with thermal energy storage (TES) and a battery energy storage system (BESS). We reformulate operational dispatch as a Markov decision process (MDP) with continuous control actions and a feasibility-projection layer that maps agent actions to physically admissible power flows. Proximal policy optimization (PPO) is adopted to learn a closed-loop dispatch policy that can react to variability in demand and renewable output. Using an 8064-hour held-out evaluation horizon, PPO with a nominal loss-of-load penalty improves the shaped return by 4.80% relative to a fixed-priority rule-based baseline, but increases loss of power supply probability (LPSP) from 3.60% to 11.17% due to near-elimination of BESS cycling. Increasing the loss-of-load penalty recovers the baseline reliability (LPSP 3.60%) and curtailment (42.25%), but causes PPO to collapse to the rule-based policy. These results highlight the sensitivity of RL dispatch to reward weights and motivate constrained/safe-RL formulations that enforce reliability targets while optimizing storage usage.
Monzer Khalid, A. Al Hanbali, M. Aldurgam et al.· International Conference on...· 0 citations
