Semantic Language Models for WCAG — data, code and supplementary analyses

Abstract

Semantic Language Models for WCAG — data, code and supplementary analyses Supplementary material for: B. O. Kuzikov, O. A. Shovkoplias, P. O. Tytov, S. R. Shovkoplias, O. V. Shutylieva, O. V. Vlasenko. A Statistical Framework for Consensus-Based Reliability Assessment in Large Language Model Evaluation Applied to Web Accessibility. ICMLBDA 2025. The archive contains the dataset of “visible text – accessible name” pairs (WCAG 2.5.3), the scores of 17 LLM raters, the analysis code, and the supplementary analyses S1–S5 referred to in the paper as “supplement [17]”. The dataset is also mirrored on Kaggle: https://www.kaggle.com/datasets/tytovpavel/semantic-language-models-for-wcag. Contents Path Description dataset.zip Dataset and per-step artefacts as released on Kaggle (folders data/, docs/: pairs, prompts, raw LLM batch outputs) docs/Step 2 LLM-as-judge/{en,uk}/*.xlsx (in zip) Score matrices: one row per pair, one column per LLM (scale [-1, 1]) docs/Step 2 LLM-as-judge/{ua_univ,top500}/*.xlsx (in zip) Scores for real pairs from Ukrainian university websites and top-500 websites icc.py ICC(2,1) and ICC(2,k) (Shrout & Fleiss, two-way random effects, absolute agreement) load.py Loading of the score matrices, model list, cores and prices (Table 1 of the paper) analysis.py Analyses S1–S4 on the empirical data → results_real.json sim.py Monte-Carlo study S5 → results_sim.json results_real.json, results_sim.json, *.csv Reference outputs used in the paper; the CSV files are the tables of this README requirements.txt, LICENSE Python dependencies; license Reproduction Download all files into one folder and unpack dataset.zip there (it creates data/ and docs/): unzip dataset.zip python -m venv venv && venv/bin/pip install -r requirements.txt venv/bin/python analysis.py "docs/Step 2 LLM-as-judge" 500 venv/bin/python sim.py 200 Re-running overwrites the reference JSON files with identical values. Random seeds are fixed (2025 for the bootstrap, 42 for the simulation). ICC is computed on items with scores from all raters of the evaluated set (listwise deletion), as in the paper. The values reported in the paper (ICC2k = 0.977 for the 9-model pool on 86.4% of items and 0.955 for the 3-model core) are reproduced exactly by analysis.py. Models: pool of 9 (Fig. 1 of the paper) = gpt-4.1, claude-3.5-haiku, deepseek-prover-v2, deepseek-r1, llama-4-maverick, gemini-2.5-flash-preview, gpt-4o-mini, qwen3-235b-a22b, gemini-2.0-flash-001; core of 3 = claude-3.5-haiku, deepseek-r1, gemini-2.5-flash-preview. Price $c_j$ = blended price per 1M tokens (3:1 input:output, OpenRouter, Table 1). S1. Proposition 2: ICC under extension of the reference set Statement. Let $\widetilde{\varepsilon}_1 \subset \widetilde{\varepsilon}_2$ be rater sets of sizes $k_1 < k_2$ evaluated on the same items, and $\lambda_1, \lambda_2 > 0$ their single-rater coefficients ICC(2,1). Then $\text{ICC}(2,k) = k\lambda / (1 + (k-1)\lambda)$ exactly, for the population parameters and for the ANOVA estimators; $\text{ICC}(2,k_2) \ge \text{ICC}(2,k_1)$ if and only if $\lambda_2 \ge \lambda^{\ast} = k_1\lambda_1 / (k_1\lambda_1 + k_2(1-\lambda_1))$; if the added raters are exchangeable with the existing ones ($\lambda_2 = \lambda_1$), the ICC strictly increases. Proof. Under $x_{ij} = \mu + r_i + c_j + e_{ij}$ with variances $\sigma_r^2, \sigma_c^2, \sigma_e^2$: $\text{ICC}(2,1) = \sigma_r^2 / (\sigma_r^2 + \sigma_c^2 + \sigma_e^2)$ and $\text{ICC}(2,k) = \sigma_r^2 / (\sigma_r^2 + (\sigma_c^2 + \sigma_e^2)/k)$; dividing numerator and denominator gives (1). For the estimators (n items, k raters), let $A = MS_R - MS_E$ and $B = (MS_C - MS_E)/n$. Then $\widehat{\text{ICC}}(2,1) = A/(A + k\,MS_E + kB)$ and $\widehat{\text{ICC}}(2,k) = A/(A + MS_E + B)$; substituting the first into $k\lambda/(1+(k-1)\lambda)$ yields $kA/(kA + k\,MS_E + kB)$ = the second. From (1), $1/\text{ICC}(2,k) = 1 + (1-\lambda)/(k\lambda)$, so $\text{ICC}(2,k_2) \ge \text{ICC}(2,k_1) \iff (1-\lambda_2)/(k_2\lambda_2) \le (1-\lambda_1)/(k_1\lambda_1)$, which rearranges to (2). (3) follows since $k_2 > k_1$. ∎ Why the original formulation was insufficient. A positive rank correlation of an added rater with the consensus does not bound its error variance or bias. Such a rater can lower $\lambda_2$ below $\lambda^{\ast}$, and the ICC then decreases. Empirical check. For the 3-model core $\lambda_1$ = 0.877, so for $k_2 = 9$: $\lambda^{\ast}$ = 0.704. Observed $\lambda_2$ (9-model pool) = 0.824 > $\lambda^{\ast}$, consistent with ICC2k rising from 0.955 to 0.977. Adding a single non-core model to the core (table below; add_one_rater.csv): all 14 models correlate positively with the core consensus, but 9 of them lower ICC2k. Added model n Spearman ρ with core ICC2k core ICC2k core + model Δ openai/gpt-4.1 11282 0.830 0.955 0.968 +0.0127 qwen/qwen3-235b-a22b 11237 0.834 0.955 0.963 +0.0071 openai/gpt-4o-mini 11162 0.820 0.955 0.961 +0.0057 meta-llama/llama-4-maverick 11241 0.804 0.955 0.960 +0.0048 deepseek/deepseek-prover-v2 11200 0.780 0.956 0.956 +0.0006 google/gemini-2.0-flash-001 11283 0.812 0.955 0.955 -0.0005 liquid/lfm-7b 5236 0.635 0.956 0.941 -0.0155 amazon/nova-micro-v1 11248 0.687 0.956 0.933 -0.0223 meta-llama/llama-3.1-8b-instruct 10163 0.621 0.954 0.915 -0.0394 openai/gpt-4.1-nano 11202 0.623 0.955 0.915 -0.0399 qwen/qwen2.5-coder-7b-instruct 10117 0.517 0.952 0.905 -0.0470 mistral/ministral-8b 10269 0.449 0.953 0.887 -0.0661 liquid/lfm-3b 3458 0.248 0.966 0.900 -0.0664 meta-llama/llama-3.2-3b-instruct 6289 0.465 0.958 0.891 -0.0671 S2. Reliability estimands ICC(2,k) (two-way random effects, absolute agreement, average of k raters) is the reliability of the mean score of k raters sampled from a population of raters. It matches the intended use: the deployed score is an aggregate of a small core, so ICC(2,3) of the core is the reliability of the score actually used (for k = 3, median and mean differ only on outlying ratings; see S5); absolute agreement penalizes systematic offsets, which matter on a fixed [-1, 1] decision scale; LLMs are heterogeneous and are regularly deprecated or replaced, so treating them as a sample from the pool of capable models describes a replaceable core rather than one fixed set of versions. Because the random-rater assumption is an idealization for heterogeneous models, the single-rater ICC(2,1) and the fixed-rater consistency coefficient ICC(3,k) are also reported. ICC(3,k) ≈ ICC(2,k) means systematic offsets between core models are negligible relative to item variance. The out-of-distribution rows use real pairs from Ukrainian university websites and top-500 websites. Reference set / data n ICC(2,1) ICC(2,k) ICC(3,k) 9-model pool, synthetic EN+UA 11071 0.824 0.977 0.977 3-model core, synthetic EN+UA 11283 0.877 0.955 0.957 9-model pool, university sites 165 0.624 0.937 0.949 3-model core, university sites 242 0.725 0.888 0.893 9-model pool, top-500 sites 246 0.571 0.923 0.936 3-model core, top-500 sites 348 0.694 0.872 0.876 S3. Circularity: leave-one-out and independent reference Each core model’s R² is computed against (a) the core median, which includes the model itself (in-sample); (b) the mean of the two other core models (leave-one-out, LOO); (c) the median of the six pool models outside the core (independent reference). Items with scores from all 9 pool models, n = 11071. Core model R² in-sample R² LOO R² independent (6 models) claude-3.5-haiku 0.89 0.80 0.86 deepseek-r1 0.88 0.81 0.87 gemini-2.5-flash-preview 0.92 0.84 0.87 Agreement of the two disjoint consensus scores (3-model core vs. 6 other pool models): Pearson r = 0.967, Spearman ρ = 0.908, mean absolute difference = 0.074, ICC(2,2) = 0.983. Non-core models do not contribute to the consensus, so Table 2 of the paper is not affected by self-inclusion. Their R² against the core consensus and against the independent 6-model reference (same items) gives the same ranking: Non-core model R² vs core R² vs 6-model reference gpt-4.1-nano -0.73 -0.56 ministral-8b -1.39 -1.18 nova-micro-v1 0.39 0.47 llama-3.1-8b-instruct 0.14 0.20 llama-3.2-3b-instruct -0.01 0.06 lfm-3b -0.51 -0.47 qwen2.5-coder-7b-instruct -0.04 0.02 lfm-7b 0.46 0.51 S4. Core selection: objective, constraints, algorithm, stopping rule, stability Problem. Pool $P$ (9 models, Fig. 1), price $c_j$, budget $C$ = 3 USD per 1M blended tokens: $$\widetilde{\varepsilon}_k^{\ast} = \arg\max_{|\widetilde{\varepsilon}| = k} \text{ICC}(2,k)(\widetilde{\varepsilon}) \quad \text{subject to} \quad \sum_{j \in \widetilde{\varepsilon}} c_j \le C \qquad (1)$$ This is Eq. (1) of the paper, with $\widetilde{\varepsilon} \subseteq P$; the subscript $k$ marks the core size used in the stopping rule. Algorithm. Exhaustive enumeration ($\binom{9}{k} \le 126$ subsets per k). All subsets are evaluated on the same items (rated by all 9 pool models, n = 11071) for comparability. Stopping rule. $k^{\ast} = \min \lbrace k : \text{ICC}(\widetilde{\varepsilon}_{k+1}^{\ast}) - \text{ICC}(\widetilde{\varepsilon}_k^{\ast}) < \delta \rbrace$, $\delta$ = 0.01, subject to $\text{ICC}(\widetilde{\varepsilon}_{k^{\ast}}^{\ast}) \ge 0.90$. The number of models (API calls, latency, provider dependency) is the primary cost; the price is the secondary constraint. The sum of $R_j^2$ (Eq. (1) of the submitted version) is not used as objective, because it grows with |E| by construction and is inflated by self-inclusion (S3). Result. Best feasible cores: k = 2 → ICC2k 0.940; k = 3 → 0.956 (claude-3.5-haiku, deepseek-r1, gemini-2.5-flash-preview, 2.78 USD); k = 4 → 0.957. The gain 3 → 4 (0.0015) is below δ, hence k* = 3 and the selected core is the one used in the paper. Full path with and without the budget: size_path.csv. k Constraint Best subset Cost, USD ICC2k 2 free gpt-4.1, gemini-2.5-flash-preview 3.7