Ujjawal K. Panchal

FunDM: Cheap&Label-Free Decision Models from Frozen Language Models

Decision models such as Jev score candidate answers or actions. A small decision head trained label-free can recover about 99% of a frozen language model's multiple-choice accuracy while using only two-thirds of its transformer layers. It learns the full model's answer probabilities with KL divergence, without correct-...