Matched data, small gaps: a pre-registered comparison of fine-tuned classifiers, retrieval-prompted language models and a typed-decision model for intent and out-of-scope decisions

Abstract

Before an assistant acts on a message, it decides whether the message is something it handles (scope) and what it asks for (intent). We ask which kind of system should make that decision given the same in-scope labelled data, and what moves a prompted model's scope decision. We compare a fine-tuned 150M classifier with retrieval-prompted 27–31B language models on three intent sets, in protocols frozen before any request; only the classifier also calibrates its scope decision on out-of-scope development items, and a hosted typed-decision model is a secondary system. On Banking77 the two are not significantly different, though equivalence within 2.5 points is not shown, and MASSIVE shows no stable ordering across four draws of held-out domains. The one significant pre-registered gap (CLINC150, 1.9 points for the language model) halves and is not significant on items we had never inspected (post hoc). The clearest finding concerns the second question. On fresh label-clean BANKING77-OOS items, retrieving in-scope examples lowered five models' rejection of same-domain out-of-scope requests by 19 to 30 points, while random examples moved it by 1 to 11. This is mainly a shift of the operating point towards acceptance, not a demonstrated loss of discrimination: at equal in-scope acceptance, the typed-decision model's rejection rose. Labelled out-of-scope examples raise rejection, mainly for the requests that retrieve them. We also report a label leak we found in BANKING77-OOS and the exploratory conclusions we withdrew.