Skip to content

Category

natural language processing

9,568 papers

#artificial intelligence Preprint Open access Oct 2026

Conceptizer: Next-Token Prediction Is Not All You Need

Next-token prediction (NTP) rewards only the token that occurs next, penalizing every alternative. But natural language often offers many ways to express the same meaning. We introduce Conceptizer, a training framework that extends next-token prediction with a reward for meaning. Conceptizer uses a fixed, model-agnosti...

Haoran Zheng, Chen Shani · 0 citations
#artificial intelligence Preprint Open access Oct 2026

Mine or Yours? Mind-Bridged Language Models Take Each Other's Memories as Their Own

If your brain were connected directly to someone else's, would you still know which thoughts were yours? We build a functional version with two copies of one language model: each holds a private rule and code word, and one copy, the receiver, also attends to its partner's internal memory. The receiver took its partner'...

Yunlong Xu · 0 citations
#artificial intelligence Preprint Oct 2026

SUDO: From Scientific Publications to Global Scholarly Discourse

Scholarly Knowledge Graphs (SKGs) support literature search, scientific discovery, and agent-assisted research. Yet, most approaches to SKG construction pre-specify the knowledge unit to be represented, typically named scholarly entities or contribution-level statements, and retain only the publication content expressi...

N. Pathirage, Henrique M. Dinis Santos, Deborah L. McGuinness · 1 citation
#machine learning Preprint Oct 2026

AdaptTok-Indic: Script-Aware Adaptive Tokenization for Indic Languages

Results suggest that Indic-specialized adaptive segmentation can reduce tokenization disparity while maintaining competitive downstream performance, and suggest that Indic-specialized adaptive segmentation can reduce tokenization disparity while maintaining competitive downstream performance.

Binayak Mukherjeea, Moitreya Nandi, Arindam Panja · 0 citations
#artificial intelligence Preprint Open access Oct 2026

Merge-Based Vocabulary Extension Stalls at the Pre-Tokenizer: An English-Preserving Repair for Nepali, Tested in Continued Pretraining

Llama-3, Qwen3 and many other open LLMs pre-tokenize text with a regular expression whose word class admits only Unicode letters. Devanagari vowel signs are combining marks, so this regex splits Nepali words at every vowel sign, and models that use it spend 2.6 to 5.8 times as many tokens on Nepali as on the same Engli...

Aarjan Chaudhary (Naamche Labs) · 0 citations
#artificial intelligence Preprint Oct 2026

SLICE-Mem: Salience-Gated Lightweight Information Curation for Efficient LLM Agent Memory

Long-term memory enables large language model (LLM) agents to maintain personalized and temporally coherent interactions, but repeated LLM inference for memory admission, extraction, and organization introduces substantial cost and latency. We present SLICE-Mem, a lightweight memory framework that combines salience-gui...

Xiao-Bin Chen, Dian Jin, Lin Shu et al. · 0 citations
#artificial intelligence Preprint Open access Oct 2026

The Social World Model Evaluation Gap

A negotiation assistant can reach a good deal without accurately predicting the other person's response. We ask whether evaluations of social world models test that prediction directly. We audit 113 works published from 1999 to 2026. 9 primary evaluations measure a real person's response to an action selected by the sy...

Denglin Jiang (New York University), Annie Wang (New York University) · 0 citations
#artificial intelligence Preprint Open access Oct 2026

Recursive Semantic Divergence for LLM Agent Consistency

Large Language Model (LLM) agents with tool use exhibit inconsistent behavior across independent runs from identical states, calling different tools, producing contradictory outputs, or pursuing divergent strategies. This variability compounds over time and undermines deployment reliability. Existing evaluation metrics...

Harshavardhan Abichandani, Penny Chong, Atin Ghosh et al. · 0 citations
#artificial intelligence Preprint Open access Oct 2026

TraceQuery: learning what to recompute from full-denoising trajectory in diffusion language models

Diffusion large language models (dLLMs) enable flexible generation through iterative masked denoising, but incur substantial inference cost because token representations are repeatedly recomputed across denoising steps. Existing cache-based methods reduce this cost by reusing intermediate results across denoising steps...

Yukai Jin, Zhankai Ye, Jiayue Yang et al. · 0 citations
#natural language process... Preprint Open access Oct 2026

Knowledge-Guided LLM for Intent-Driven Service Orchestration in Integrated Terrestrial and Non-Terrestrial 6G Networks

Natural-language service intents in integrated terrestrial and non-terrestrial (TN/NTN) 6G networks can be incomplete, ambiguous, or inconsistent, making unconstrained LLM generation challenging for reliable network-facing orchestration. This paper proposes a knowledge-guided LLM framework that combines grammar-constra...

Wajdi Hammami, Irfan Azam, Soumaya Cherkaoui · 0 citations
#machine learning Preprint Open access Oct 2026

Linear-Time Context Compression with State-Space Encoding

Transformer-based large language models (LLMs) impose substantial computation and memory costs as context sizes increase, motivating methods that compress a context into a shorter representation before inference. Existing soft-prompt compression methods typically use the same Transformer backbone as used in the target...

Marc Serra Ortega, Andrey Barsky, Carlos Boned Riera et al. · 0 citations
#machine learning Preprint Open access Oct 2026

OpenESM: Energy-Steered Models Are Scalable Foundation Learners

Energy-based models have shown promising properties for domain-specific tasks but remain challenging in modeling general-purpose scenarios. To bridge this gap, we design energy-steered models (ESMs) for foundational language modeling. ESMs integrate energy steering and token generation within a single architecture to l...

Yuan Pu, Yudong Lu, Ting Wang et al. · 0 citations

From tech blogs

See all →
Microsoft Research Blog Oct 22, 2026

Flint-interactive: A language for creating interactive data visualizations

Flint-interactive gives AI agents a language for creating interactive visualizations. Click a chart, filter data, highlight related content, and explore beyond the view the author originally chose. The post Flint-interactive: A language for creating interactive data visualizations appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.