SupraTok: Cross-Boundary Tokenization for Enhanced Language Model Performance
Results show that crossing whitespace boundaries gives consistent compression gains under controlled public-data comparisons, while optional entropy filtering provides a separate pipeline benefit.
Andrei-Valentin Tanase, Elena Pelican
· Transactions of the Associat... · 3 citations
· ⚡1