Conservative Reference-Based Judging of AI-Generated STEM Analogies Using an Open Language Model
Abstract
Generative AI can rapidly produce analogies for STEM teaching, but fluent comparisons may encode false relations or obscure boundary conditions. This study evaluates whether a locally served open LLM, Qwen3-14B, can act as a conservative reference-grounded screen for school-level analogies. The judge assessed 376 public English and Hindi items under three decision policies, yielding 1,128 parseable calls. It rejected every planted wrong probe, although the probe set represented only five unique misconception structures and the Hindi probes were direct translations. Across tiers, all good and decent probes were accepted, 29 of 30 vague observations were routed away from acceptance, and all 30 wrong observations were rejected. For 256 ordinary English generations, source-domain scores differed overall, but corrected pairwise tests supported only limited contrasts, mainly lower scores for cooking analogies. Against original-panel silver labels, an English TF–IDF and metadata model achieved macro-F1 of 0.706 with item-level cross-validation and 0.661 with concept-grouped folds, while the best English Qwen policy reached 0.632. The evidence supports open-LLM screening for explicit reference contradictions, but broader claims about instructional quality remain constrained by policy choice, language scope, translated probes, surface-predictable labels, and the absence of teacher ratings or learner outcomes.
