A central hope behind process supervision is that models can expose intermediate variables that matter for their later behavior. For this to help with alignment, a scratchpad must be tied to the computation: when the model writes a state, later steps should compute from that state. To test this requirement, we use a controlled state-tracking task with a known update rule, comparing models trained to report only the final state with models trained to write intermediate states before giving the final answer. At evaluation, we edit the internal representation of one written state while leaving the visible scratchpad text fixed. Because the transition rule is known, the edit has a single correct downstream consequence. In Qwen2.5-Coder-7B, the state-writing model predicts the next phase bit implied by the edited state on 80% and 91% of held-out examples across the two task variants, while pretrained and final-answer-only controls remain near baseline. Additional controls rule out generic next-token steering and copying another continuation: the prediction depends on both the edited state and the current move. The same causal-use pattern replicates across model families. Together, these results suggest a sharper goal for scratchpad oversight: not just to make intermediate reasoning legible, but to train written states that the model uses as part of its computation.
Benjamin Shih, John Winnicki, Eric Darve· arXiv.org· 0 citations
A novel committor learning framework grounded in the AlphaFold 3 paradigm is proposed that elucidates how ligand substituents regulate the ratio between distinct binding pathways, offering new perspectives for structure-based drug design.
Jintu Zhang, Zichang Jin, Huifeng Zhao et al.· 0 citations
MemDefrag, a training-free and model-agnostic framework that uses a middle-layer tracing signal to conduct memory defragmentation (rank, reorder, and filter memories), and applies an informativeness-guided proportional forgetting mechanism once capacity is exceeded, is proposed.
This work uses a full $2^3$ factorial design to decompose three recurring interventions in formalization pipelines: parametric expert drafting, Mathlib/context search, and Lean elaboration feedback, suggesting that formal validity, proof-oriented Lean competence, and faithful statement generation should be reported separately.
Ke Zhang, P. Gallardo, S. Murthy et al.· arXiv.org· 1 citation
Reach audiences
Advertise in front of researchers, engineers, and readers.
EmoTrace, a multi-turn dialogue corpus generation framework centered on modeling seekers' emotional trajectories, is proposed, which outperforms existing approaches in terms of emotional richness and empathy quality.
Kai-Tong Weng, Lixiong Liu, Zihao Liu et al.· arXiv.org· 0 citations
Gaokerena, a novel family of compact Persian medical language models optimized for deployment on consumer grade hardware, and Gaokerena-R, a novel family of compact Persian medical language models optimized for deployment on consumer grade hardware, are presented.
MameLoshnLM, the first open-source 8B-parameter language model built specifically for Yiddish, is presented, providing both a foundation for Yiddish NLP and a practical template for language model development in historically rich but digitally underrepresented languages.
Uri Katz, Omer Goldman, Tomasz Limisiewicz et al.· 0 citations
A large-scale empirical study across a diverse set of embedding models and 275+ languages spanning three parallel datasets, exposing persistent gaps in cross-lingual semantic representation that track language prevalence in training resources and subword tokenization.
Andrianos Michail, Stylianos Psychias, Michelle Wastl et al.· 0 citations
LookBack, a training-free LVLM response scoring method that augments token likelihood with visual lookback score, a lightweight measure of how strongly each response token refers to image tokens, consistently improves Best-of-$N$ selection over existing baselines with negligible additional overhead.
Beomsik Cho, Jin-Ha Kim, Dongseok Lee et al.· 0 citations
This paper introduces Caduceus, a family of MoE-enhanced foundation models built with a hierarchical pre-training paradigm to jointly integrate biological and natural language, and incorporates a multi-task instruction tuning phase, enabling robust protein parsing and natural language question answering.
Mingze Yin, Yiheng Zhu, Jialu Wu et al.· Proceedings of the 32nd ACM...· 0 citations
The null token is turned into a diagnostic lens on hallucination and motivate evaluating abstention methods by both suppression and deletion costs, rather than by hallucination reduction alone.
Kirill Borodin, V. Kudryavtsev, Ivan Viakhirev et al.· 0 citations
Scaling transformer language models creates an inherent tension between expressivity and memory efficiency. While unique weights across layers preserve functional specialization---from input-grounding to abstract refinement---they incur a substantial memory footprint. Conversely, standard depth-sharing enforces uniform transformations that collapse representational diversity and degrade modeling quality. We introduce Gated Recurrent Transformer, a recurrent depth transformer where fixed-depth prelude and coda blocks bracket a single shared core iterated R times. Inspired by gated recurrent neural networks, we employ a lightweight projection and an elementwise update gate---conditioned on the hidden state, the fixed prelude output, and noise resampled at every step---to modulate the recurrent update. This allows the model to specialize the input to the same few layers across recurrences, rather than requiring many unique layers to achieve functional diversity. Under an isoFLOPS constraint, a 3-layer Gated Recurrent Transformer matches the accuracy of a 12-layer GPT-2 Small baseline with similar training and inference FLOPs, and leads MoR and heavy-tail depth sampling in all nine scale-by-budget cells; at medium and large scale it approaches dense quality at the standard token budget and overtakes it at medium scale once that budget is doubled. Under an isoPARAMS constraint, deeper recurrence achieves a 2.76 validation loss versus 2.84 for a non-recurrent counterpart at matched parameter and data budget. Our results demonstrate that adaptive depth reuse is a principled strategy for trading parameters for quality: at large scale, 63% fewer parameters and 59% less peak decoding memory for a 10% increase in compiled generation latency.
Amr Hegazy, Amr Alanwar, Mostafa Elhoushi· 0 citations
A new method, called CW-Net, translates the reasoning process of an autonomous vehicle’s AI system into understandable concepts that explain its behavior.
MIT News · Artificial Intelligence· news.mit.eduAug 31, 2026
With millions of users across the world, Julia has been used to conduct cutting-edge research and to design new drugs, jet engines, heat pumps, and more.
A new machine-learning framework aims to improve the success rate of computational protein design while moving away from results that reproduce sequences found in nature.
New MIT research could lead to better materials for a fossil-fuel-free process for making the chemical that's essential to fertilizer and other products.
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
