Deep Reinforcement Learning-Based Energy Management of a Microgrid With Nonlinear Battery Losses and Degradation

Abstract

In microgrids, the energy management system (EMS) manages distributed energy resources (DERs) to achieve cost-effective operation under time-varying tariffs. However, the uncertain nature of load demand and renewable generation limits the applicability of model-based approaches, and the nonlinear losses and degradation of the battery energy storage system (BESS) further complicate the dispatch problem. Deep reinforcement learning (DRL) offers a model-free alternative, but the combination of these nonlinearities and uncertain conditions poses an exploration challenge for the learning agent. In this paper, we propose a soft actor-critic (SAC)-based EMS algorithm for a grid-connected microgrid with photovoltaic (PV) generation, a microturbine (MT), and a BESS modeled with nonlinear charging and discharging losses and cyclic degradation. The entropy maximization of SAC maintains policy stochasticity and mitigates premature convergence to suboptimal policies. We also introduce a reward formulation that excludes the action-independent term to reduce the reward variance. Simulation results with real-world data under both time-of-use (ToU) and real-time pricing (RTP) tariffs show that the proposed algorithm achieves the lowest total operational cost among the competing benchmarks and approaches the optimization-based reference that has access to day-ahead foresight. The advantage over deterministic-policy benchmarks widens as price uncertainty increases under RTP. An ablation study further shows that both the BESS model and the reward formulation decrease the total operational cost under both tariffs.