CALM-TS: Risk-Controlled LLM Labeling for Time-Series via Calibrated Selective Gating
Labeling time-series events such as anomalies and system failures is expensive and subjective, and distribution drift makes label definitions evolve over time. Large language models can generate weak labels with natural-language rationales, but their error rates are unbounded and their failure modes opaque. We present CALM-TS, a risk-controlled weak-supervision pipeline that bounds the labeling error rate while maximizing coverage. CALM-TS introduces behavioral probing over prompt surface form, sampling temperature, and temporal context window to elicit disagreement signals; a lightweight calibrator converts them into risk-bounded acceptance with finite-sample guarantees. CALM-TS attains 77--81% cost reduction on MIMIC-III and Yahoo~S5 at empirical risk ≤ a=0.05 in all 10 dataset-seed pairs. On the official PhysioNet Challenge 2015 binary alarm-verification task with gpt-4o-mini over five seeds, CALM-TS is the only method among nine LLM-as-weak-labeler baselines whose 5-seed mean risk strictly falls below the unfiltered LLM at non-trivial coverage, delivering a 15.0% relative reduction (0.400 → 0.340) at coverage 0.286. The framework yields auditable evidence chains and a coverage-based, label-free drift indicator.