Prediction and Entropy of Evolving Sequences: A Transformer-Based Estimator of Functional Information
To resist thermodynamic disorder, living systems maintain low entropy genomic configurations through selection. We propose pseudo-perplexity, the per position surprisal of a masked language model, as an estimator of this biological functional information. Using a unified framework, we demonstrate that transformer deriv...