Probe, Don't Prompt: A Hidden-State Probe for Metadata Filtering in Multi-Meta-RAG

Abstract

Multi-Meta-RAG improves retrieval for multi-hop question answering by filtering a vector store on metadata, specifically the news source and the publication date, which it extracts from each query using GPT-3.5. This extractor requires a proprietary API call for every query, and its output is not restricted to the allowed sources: even though the prompt specifies 49 sources, GPT-3.5 generates 128 distinct source names, 79 of which do not appear on the list. We show that the source part of this extractor can be replaced by a local, deterministic probe trained on the hidden states of a small open-source language model. The probe is a multi-label logistic classifier on the mean-pooled hidden states of one layer, trained with a class-balanced loss on the gold evidence sources. Using 5-fold cross-validation to select the layer, we evaluate four models ranging from 135M to 1.5B parameters out-of-fold across all 2556 MultiHop-RAG queries. The probe reaches a set-exact accuracy of 90.9%, compared to 88.0% for a substring baseline and 80.9% for GPT-3.5. The difference comes from null queries, for which GPT-3.5 never predicts an empty source set. Across the remaining queries, all three methods achieve set-exact accuracies within 1.1 percentage points of each other. Unlike earlier probing work, we use the probe as a component of a retrieval pipeline rather than for interpretation. For all four models, the cross-validation selects one of the first layers, in contrast to the middle layers reported for retrieval tasks. We attribute this to the source name appearing verbatim in 95.4% of gold query-source pairs. The 135M model is 1.5 percentage points less accurate than the 1.5B model, so the filter requires only a partial forward pass through a small model and one linear layer. The code is available at https://github.com/mxpoliakov/Multi-Meta-RAG.