Semantic Self-Distillation for Language Model Uncertainty
It is shown that sampled semantic distributions can be distilled into lightweight student models which estimate a prompt-conditioned density before the language model generates an answer token, which can serve as a general framework for distilling predictive uncertainty in complex output spaces beyond language.