LLM Board Game Tournaments: A Dataset of Moves, Validity, and Reasoning

Abstract

We present A Dataset of LLM Strategic Play , comprising 9,572 tournament records from a controlled eight-round Swiss-system tournament involving 20 LLMs across four board games: Tic-Tac-Toe, Connect Four, and their goal-inverted variants, Suicide and Not Connect Four. Move records include the model identity, board state, legal moves, observed latency, and textual justification. Moves are validated with a GDL-based arbiter and a rectification protocol that allows up to 10 retries. Overall, 25.2% of LLM-generated moves are invalid, approximately 50% in Suicide; in total, 106 game sequences ended in automatic forfeit, represented by terminal records in the dataset. Even the best-performing model records an invalid-move rate of 10.7%, showing that rule-following errors remain present under these evaluation conditions. These observations illustrate the use of simple board games as controlled domains for evaluating LLM behavior, where strategic decision-making can be examined together with compliance with formal constraints. The proposed pipeline–GDL-based validation, Swiss tournament design, and rectification protocol–is domain-agnostic and can be extended to other GGP settings, supporting scalable evaluation of LLM decision-making under formal rules.