Bridging Chess Engines and Language Models: A Novel Framework for Generating Human-Like Chess Commentary

Abstract

This paper proposes a reinforcement learning (RL)-based framework that automates chess commentary generation using real-time board analysis. It combines chess engine evaluation along with natural language generation to produce strategically grounded commentary which is also linguistically fluent. The framework consists of three major stages: 1) a board-state evaluation layer that makes use of Stockfish engine outputs to analyse positions, 2) a structured annotation layer based on Proximal Policy Optimisation (PPO) that generates quality, phase, and tactical commentary tokens, and 3) a language generation layer utilizing leading Large Language Models (LLMs) such as GPT-4o, Gemini 2.5 Pro, and Gemma 4 to synthesize natural language commentary. A seven-metric automated evaluation framework was developed that incorporates GPT-2 perplexity, DeBERTa-v3 Natural Language Inference (NLI), Sentence-BERT similarity, MTLD and Self-BLEU to evaluate the generated commentary. Experiments on championship-level games show that the framework balances factual consistency, strategic depth, and natural language fluency quite effectively, while revealing measurable trade-offs between various LLMs. The proposed system demonstrates the potential of combining RL with generative AI for application purposes in live chess broadcasting, educational platforms, and intelligent game analysis systems.