Preserving LLM Output Distributions for Probabilistic Query Evaluation
Abstract
Language models can be used to augment databases; typically, when this is done, their top prediction is used while the uncertainty in this prediction is ignored. We propose a framework that preserves output distributions from language model predictions and uses them to evaluate SQL queries probabilistically. First, a language model is used to infer new features from existing ones. Second, ProvSQL, a PostgreSQL extension implementing a probabilistic database through data provenance, encodes the predictions as mutually exclusive events in a provenance circuit and evaluates probabilistic queries. We extend ProvSQL to support text-value comparisons and join operations on language model predictions. We demonstrate the framework on 10\,612 Amazon musical products and 230\,156 user reviews, locally augmented with the Llama 3.1 model. We show that probabilistic querying reveals contradictions between model predictions and existing labels, recovers data records otherwise missed, and propagates joint uncertainty across multi-table queries.