Retrofitting language models to operate over bytes.
Recent advances in artificial intelligence (AI) have largely been driven by large language models, deep neural networks that operate over discrete units called tokens. To represent text, most large language models use words or word fragments as the tokens, known as subword tokenization1. Subword tokenization obscures f...
Benjamin Minixhofer, Tyler Murray, Tomasz Limisiewicz et al.
· Nature · 1 citation