Leaked Labels, Masked Minimal Pairs and Normalised Inputs: Research Package for Testing Claude Code Agents on Kazakhstan's Bilingual Criminal Code

Abstract

This repository contains Supplementary File S1 for the manuscript “Leaked Labels, Masked Minimal Pairs and Normalised Inputs: Testing Claude Code Agents on Kazakhstan’s Bilingual Criminal Code”, prepared for the journal AI (MDPI). The research examines the reliability of Claude Haiku, Sonnet and Opus 5.5 agents in detecting formally defined changes in Russian and Kazakh legal texts. The repository includes: • Russian and Kazakh source texts of Kazakhstan’s Criminal Code, with provenance records and checksums.• A benchmark of 680 rule-generated cases, including 80 pilot and 600 main cases.• Formal edit rules and an independent implementation of the validation oracle.• Experimental protocols, prompts, model responses, run transcripts and audit records.• Follow-up experiments F1–F4.• Confirmatory protocol F5 containing 976 masked minimal-pair items.• Control protocol F6 examining the order of response fields.• Evaluation tables, statistical analyses and figures.• Post hoc human verification materials.• Python scripts and tests for offline reproduction. The main benchmark revealed label leakage through the surface form of edits. The confirmatory experiments provide evidence of value comparison for simple masked changes, subject to residual numeric cues and protocol limitations. The repository supports independent verification and reproduction of the reported computational results. Instructions are provided in README.md and DATA_DICTIONARY.md. The reproduction script is scripts/reproduce.py. Licensing: original software is distributed under the MIT License; generated research data and documentation are distributed under CC BY 4.0. Official legal source texts are treated separately, as explained in LICENSES.md. The artificially edited legal fragments are research materials, not official legislation, and must not be used as legal advice.