Hybrid Genetic Programming and Q-Learning for Behavior Tree Evolution inMicroRTS Tactical Scenarios

Abstract

Genetic Programming (GP) can be used to evolve human-interpretable Behavior Tree controllers for realtimestrategy games. Current Behavior Trees-based GP approaches compute fitness only at the end ofeach episode, which does not allow learning from finer-grained tactical decisions during gameplay. In thiswork, we integrate tabular Q-learning within the BT controller to gate actions at terminal nodes based onlearned action-values, and collect additional rewards at each tick of the game that are used to augmentthe final fitness signal. Since the Q-table is reset at the start of each episode, our approach allows thelearned information to affect fitness, while keeping the learned values from one generation separate fromsubsequent generations (a form of Baldwinian learning). Applied to plain-terrain MicroRTS micromanagementchallenge (population of 100 BTs, 2000 generations, against a deterministic rush opponent), ourapproach achieves a maximum fitness of 26.44 compared to GP’s max fitness of 22.0, a relative improvementof 20.2%. Mean best-fitness was also increased by 17.8% compared to the GP-only baseline. Theseresults are reported from a single representative run per configuration; multi-seed replication is identifiedas a priority for future work. The additional per-episode RL reward signal is strongly correlated with elitefitness (r = 0.89), confirming that it provides informative guidance for the evaluation. Unit coordinationexhibited by the hybrid agents is also found to be more structured, with defined front-line and support roles.The agent maintains its interpretable Behavior Tree structure.