Retrieval-Grounded, Distractor-Robust Quantitative Reasoning with a Local Small Language Model: A System Report for DocSem 2026

Abstract

We describe a three-stage pipeline submitted to the DocSem 2026 shared task on document-grounded quantitative reasoning with evidence attribution. Each task instance provides a scanned, image-only PDF containing a self-contained arithmetic word problem embedded among a large number of lexically similar but irrelevant distractor passages, together with a heavily templated, low-information user query. Our system (i) performs OCR-based document parsing with error-tolerant block-identifier recovery, (ii) locates the evidence-bearing block using a small set of deterministic discourse-marker regular expressions rather than dense or semantic retrieval, and (iii) solves the located problem with a local 3B-parameter instruction-tuned language model, comparing free-form chain-of-thought self-consistency against program-aided (code-execution) solving. On a held-out development sample of 40 instances, program-aided solving reached 70% answer accuracy against 47.5% for chain-of-thought self-consistency, while evidence-block retrieval reached 100% across every development sample we evaluated (n > 60). We report several empirical findings of independent interest: (a) conclusions drawn from small evaluation samples (n=15) were reversed by a larger sample (n=40), underscoring the risk of underpowered pipeline comparisons; (b) the held-out test split used a materially different block-identifier format and evidence-marker vocabulary than the training and validation splits, which we interpret as a deliberate probe for retrieval logic that has overfit to surface patterns rather than task semantics; and (c) compute constraints on free consumer hardware limited our final test submission to 125 of 1,730 instances, and, on direct inspection of the submitted file, the resulting 0.00% official test score appears attributable primarily to exact-match evidence scoring failing against OCR-corrupted short alphanumeric identifiers rather than to wholesale pipeline or model failure, since 96% of completed instances received substantive (non-placeholder) answers. We discuss this outcome honestly, including the evidence supporting that diagnosis, and argue that both the operational lesson, inference throughput under a fixed compute budget is as consequential as per-instance accuracy, and the scoring-sensitivity finding, exact-match evidence metrics may be a poor discriminator of system quality when evidence identifiers are OCR-derived, are useful contributions independent of our final rank.