Alexander C. Berg

Repurposing Pre-trained LLMs as High Fidelity Continuous Text Autoencoders

LLMAE, a method for repurposing a pretrained decoder-only language model as a continuous text autoencoder by exposing the activations of an intermediate layer as a fixed-length latent bottleneck, is proposed and provides an effective substrate for downstream adaptation while benefiting from the fluency of the original...