LLM-as-Judge for OCR Quality Assessment in Digital Libraries: A Reference-Free Approach

Abstract

Optical character recognition (OCR) quality assessment has depended on reference-based metrics (CER/WER), which require costly transcription. We ask: can an LLM supply a reference-free, text-only signal for OCR readability? Using 90 stratified passages from 53 scans of 17 English works in RETAS, we score OCR text with Claude Sonnet 4.6. Scores correlate with CER (Spearman ρ = −0.55, − 0.60) and WER (− 0.65, − 0.69); Pearson coefficients are weaker. Prompts rank passages almost identically (ρ = 0.92). A trivial OOV baseline reaches ρ = 0.49 against CER and 0.61 against WER; paired bootstrap intervals for the LLM advantage include zero. Inspecting seven highest-CER passages without page images, we found one reference-side mismatch and six mixing real errors with irregularities. We treat disagreement as an audit trigger rather than evidence that CER is wrong.