Contemporary Genomic Foundation Models (GFMs) rely on a DNA-as-a-string paradigm that employs masked token prediction objectives for pretraining. However, this abstraction does not explicitly model the biochemical, structural, and physical properties essential to biological function. Many molecular properties can be es...
Jeremy Levy, Ariel Larey, Yury Nahshan et al.· 0 citations
Self-supervised pre-training of time series models is currently dominated by next-token prediction and reconstruction objectives. In continuous-valued domains, these paradigms often waste model capacity on high-frequency, point-wise noise at the expense of learning invariant structure. While invariance-based self-disti...
A reference-aligned diploid encoding for single-nucleotide variants (SNVs) and short insertions and deletions (indels) and unphased and phase-retaining tokenizers that accept phased genotypes and convert them to single-sequence diploid representation are introduced.
Guy Leib, T. Zinger, Dan Ofer et al.· bioRxiv· 1 citation
This work introduces token-error supervision for sparse routing in two forms, one of which directly aligns the router's affinities to the model's objective without requiring an additional head or inference-time modification and both mechanisms preserve the native sparse execution budget and aggregation policy.
Yury Nahshan, Nati Daniel, Jacob Goldberger et al.· arXiv.org· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.