ChessFraud: Exploring the Capabilities of Human-Aligned Models for Cheating Detection in Online Chess

Abstract

Human-aligned chess models, designed to mimic human decision-making rather than maximize engine strength, pose a novel challenge for online fair-play enforcement. While prior work assesses these models on move prediction accuracy, their potential as sophisticated cheating tools and their utility for cheating detection remain underexplored. We introduce ChessFraud, the first public benchmark for move-level cheating detection, providing 505 tournament games with ground-truth annotations from a controlled environment where engine usage was explicitly logged. To address the scarcity of real-world cheating data, we construct ChessFraud-Synth, a large-scale synthetic dataset derived from 12,000 Lichess games. For each game prefix, we generate a paired sample by replacing the final move with a suggestion from a classical engine or a human-aligned model, creating balanced fair/cheating instances that isolate positional context from move choice. Using these synthetic data, we train detectors based on frozen representations of human-aligned models, evaluating them on both synthetic and real tournament benchmarks. Our results demonstrate that while simple engine-agreement heuristics remain a strong baseline, human-aligned embeddings provide complementary signal, enabling the training of an applicable cheating detector. ChessFraud establishes a foundational benchmark for studying cheating detection in online chess as well as the dual role of human-AI alignment in chess.