A Dual-Teacher Distilled MoE Agent for Complex Industrial Document Analysis

Abstract

Consistency verification of drilling reports is critical for engineering quality control because a single data item may be distributed across reports with different formats, units, and page structures. Existing retrieval-augmented generation methods remain sensitive to retrieval and parsing errors in such documents, whereas ultra-large models impose substantial local computing and memory costs. This study proposes a lightweight tool-augmented framework based on dual-teacher distillation and sparse mixture-of-experts (MoE) modeling. Qwen3-235B-A22B serves as the primary teacher and Qwen3-30B-A3B as the assistant teacher. Their tool-use and task-planning capabilities are transferred to a sparse MoE student upgraded from a Qwen3-1.7B dense backbone through trajectory pruning, sample decomposition, and token-level Kullback–Leibler (KL) distillation. The student adopts an eight-expert Top-2 routing architecture. Experiments on 1000 drilling reports containing 30,127 verification instances show an F1 score of 58.0 ± 0.5%, with file-level, location-level, and exact-match accuracies of 66.5%, 55.2%, and 45.0%, respectively. The model contains 9.1B total parameters and 2.8B activated parameters, and reaches a latency of 12.1 ms per forward pass and a memory footprint of 18.4 GB under bfloat16 (BF16) precision. The reported F1 score characterizes the end-to-end verification task rather than an autonomous safety decision capability. The framework is intended to support evidence localization, anomaly prioritization, and expert review in local deployment settings.