Retrieval over semi-structured knowledge bases (SKBs) must handle both textual descriptions and relational constraints. Text-based retrievers capture the overall semantics of a query but struggle with multi-hop relations, whereas graph-based retrievers follow relational paths precisely but miss the broader meaning. We...
Yicheng Tao, Yiqun Wang, Xiangchen Song et al.· 0 citations
Results indicate that strong benchmark performance of a single channel does not transfer to realistic use, and that multimodal fusion provides the robustness required of a tool that supports psychological evaluation.
Rotary Positional Embedding (RoPE) is widely used in transformers to encode relative positional information, yet the effect of applying RoPE to only a subset of hidden dimensions remains underexplored. This design choice can substantially reduce memory usage, especially at long context lengths. We show that partial RoP...
Mohammad Aflah Khan, Krishna P. Gummadi, Manish Gupta et al.· 0 citations
AlignSAE is introduced, a method that aligns SAE features with a predefined ontology through a "pre-train, then post-train"curriculum and achieves disentanglement improvements of $9$--$\infty\times$ over standard SAEs, and further supports multi-hop reasoning and a mechanistic probe of grokking-like generalization dyna...
Minglai Yang, Xin-Yu Guo, Zheng-Liang Shi et al.· arXiv.org· 4 citations
Reach audiences
Advertise in front of researchers, engineers, and readers.
Reinforcement Learning with Verifiable Reward (RLVR) effectively solves complex tasks but demands extremely long context lengths during training, leading to substantial computational costs. While multi-stage training can partially mitigate this, starting with overly short contexts often causes irreversible performance...
The vocabulary of a language model is its native semantics: the embedding and the unembedding fix the meaning of a state at the two ends of the residual stream. Between them, existing methods either reuse a fixed lens or treat the states as a point cloud, and neither reads a state the way the model does. We argue that...
Jian Gu, Hongyu Zhang, Chunyang Chen et al.· 0 citations
This work proposes novelty patching, motivated by LZ77-style compression: start a patch where the upcoming bytes are unlikely to repeat the recent context, fitted in seconds on raw text with no labels, reference model or losses.
This work trains small causal predictors on matched FineWeb-2 text and evaluates on FLORES-200 under UTF-8, UBE, and UBE's same-length big-endian variant, finding that UBE's mean error falls below UTF-8's final mean error after only 25% and 37.5% of training.
Corporate financial disclosures often unfold over time, with announcements, prepared remarks, and discussion spanning multiple high-frequency return intervals. Financial NLP benchmarks can therefore suffer from a within-event timing mismatch when the prediction target begins before the textual or audio input used by a...
Xiaoyu Ma, Kelvin J. L. Koa, Yuncong Liu et al.· 0 citations
Finetuning a language model on narrowly misaligned data can make it broadly misaligned. Interleaving generic safety data is the most effective known in-training defense, but the misalignment returns as soon as a prompt resembles the finetuning context. We argue that this conditional misalignment is a spurious correlati...
Device-cloud speculative decoding reduces server computation, but growing a deep draft tree with a small language model (SLM) requires repeated passes on a mobile neural processing unit (NPU). We present HyCo, which assigns depth to the server model's native multi-token prediction (MTP) head, executed on the device, an...
Hanbin Bae, Heejin Choi, Joohyung Lee et al.· 0 citations
Distribution shifts pose a fundamental challenge to robust multimodal learning. Existing methods typically promote invariance by suppressing environment-dependent variations, implicitly applying a uniform treatment to shifted information. However, shifted information can have different predictive consequences: some con...
Flint-interactive gives AI agents a language for creating interactive visualizations. Click a chart, filter data, highlight related content, and explore beyond the view the author originally chose. The post Flint-interactive: A language for creating interactive data visualizations appeared first on Microsoft Research.
