Reinforcement Learning-Based Energy Management for Aggregators Under Network Constraints

Abstract

High penetration of distributed energy resources(DERs), particularly residential solar photovoltaic(PV) systems and battery energy storage systems(BESS), introduces operational challenges in low-voltage distribution networks, including voltage fluctuations, peak-demand issues, and underutilization of renewable energy. The aggregation and coordinated optimization of DERs provide opportunities to improve economic performance and operational flexibility.This paper proposes a network-constrained reinforcement learning(RL)-based energy management framework for the coordinated optimization of multiple residential PV-BESS systems through a centralized aggregator within a distribution feeder. A Proximal Policy Optimization (PPO)-based RL agent is developed to optimize battery charging and discharging decisions using system states such as load demand, PV generation, and battery state of charge. Network constraints, particularly feeder voltage limits, are incorporated into the RL environment through OpenDSS-based power flow analysis. A multi-objective reward function is formulated to minimize electricity cost, reduce peak demand, and enhance renewable energy utilization while maintaining network operating limits.As inputs to the proposed RL energy management system, forecasting models for solar irradiance and electrical load are developed using Bidirectional Long Short-Term Memory (BiLSTM) and CNN-BiLSTM-attention architectures.Simulation results demonstrate that the proposed framework effectively reduces peak demand, improves voltage regulation, and increases renewable energy utilization in low-voltage feeders with high DER penetration.