This work proposes NONTP, extending NTP's signal coverage along both dimensions through two auxiliary objectives, and analyses each component contributes independently, with gradient conflict analyzed as a direction for future work.
Abstract
Next-Token Prediction (NTP) carries two structural training signal limitations. First, NTP optimizes for single-step prediction only, placing no supervised pressure on learning longer-range behavioral structure -- we term this \textbf{temporal locality}. Second, in multi-domain sequences, each target item embedding receives gradient updates exclusively from the immediately preceding hidden state, with no explicit gradient pathway from cross-domain context -- we term this \textbf{spatial locality}. We propose \textbf{NONTP}, extending NTP's signal coverage along both dimensions through two auxiliary objectives. \textbf{TCL (Temporal Contrastive Learning)} uses a BYOL-style EMA teacher with InfoNCE to align hidden states against a $K$-step future trajectory in representation space. \textbf{TDL (Trans-Domain Learning)} mean-pools cross-domain hidden states and predicts through the shared prediction head, opening a second gradient pathway with no additional parameters. Both are discarded at inference: zero overhead. On a four-domain Meituan industrial dataset (full ranking), NONTP achieves HR@10 +34.3\% over NTP and +18.3\% over MBGR. On the public Amazon Movie-Book-CDs benchmark, HR@10 +2.8\% and NDCG@10 +3.7\%. Online A/B tests confirm CTR +1.8\% and GMV +2.1\% (both $p<0.01$). Ablation studies confirm each component contributes independently, with gradient conflict analyzed as a direction for future work.
AdaMTP is proposed, an adaptive training paradigm that dynamically aligns the prediction horizon with the intrinsic predictability of the sequence, and consistently outperforms standard MTP in both task performance and inference speedup.
Ziqiang Cui, Han Shi, Bowei He et al.· 0 citations
This work proposes TTCD, a TTT framework that introduces a self-supervised objective for allocating limited memory capacity for future use, and focuses on an in-place variant: In-Place TTCD, which uses the existing MLP parameters as the fast weights.
Zixuan Wang, Xingyu Dang, Ruiming Zhu et al.· 0 citations
Large language model-based recommender systems (LLM-RSs) have demonstrated remarkable capabilities, but are computationally unsustainable for many real-world applications. Compact LLMs offer a practical alternative, yet their reduced capacity often requires reasoning or knowledge distillation methods that increase latency or depend on larger models. Combined with autoregressive generation, these approaches face severe scalability bottlenecks. In contrast, discriminative LLM-RSs enable efficient full-corpus ranking through embedding similarity, but compact backbones remain limited in expressiveness and structural adaptivity. We propose the Fusion of Layer-wise Exits for Sequential Recommendation (FLEXRec), a discriminative framework that enhances compact LLMs while retaining scalable full-corpus ranking. FLEXRec inserts prediction heads (i.e., exits) at multiple transformer layers and adaptively fuses their score distributions. An adaptive continuous router (AC-Router) dynamically selects both the number and identity of exits for each user sequence, while a novel target-k hinge loss regulates routing sparsity. Experiments on three real-world datasets with Qwen 3 1.7B and Llama 3.2 3B show that FLEXRec achieves state-of-the-art accuracy among competing methods while remaining highly efficient. Code: https://github.com/xurong-liang/FLEXRec
Xurong Liang, Tong Chen, Q. Nguyen et al.· 0 citations
Large Language Models (LLMs) have recently been explored for next Point-of-Interest (POI) recommendation. Despite progress, existing approaches face three fundamental challenges: (i) POIs are often represented by simple identifiers or categorical labels, overlooking rich textual semantics; (ii) The task of predicting the next POI is inherently sequential and context-dependent, requiring models to reason over user histories, temporal dynamics, and environmental factors; (iii) Supervised fine-tuning provides only a single predicted POI, ignoring the capacity of LLMs to generate $k$ POIs candidates. To address these issues, we propose H-RLPOI, Hybrid LLM and Reinforcement Learning Framework for next POI Recommendation, that enhances LLM representations by injecting semantic POI embeddings through token-level alignment and applies reinforcement learning with Proximal Policy Optimization (PPO) as a decision layer to optimize POI selection conditioned on user trajectories. Experiments on real-world datasets show that H-RLPOI provides context-aware, semantically grounded, and adaptive recommendations, achieving competitive or stateof-the-art performance depending on the dataset.
Zahra Hamdani, Saloua Zammali, S. Yahia· Annual International Compute...· 0 citations
While standard Next-Token Prediction (NTP) lays the foundation of language model pre- training, its teacher-forced training paradigm may not be optimal for long-horizon reasoning and planning. Recent works such as Multi-Token Prediction (MTP) and Next-Latent prediction (NextLat) try to mitigate the problem through predicting multiple future tokens and self-supervised prediction in the latent space. However, those auxiliary objectives either have a limited horizon or suffer from compounding error from multi-step rollout. We introduce Hierarchical Latent Prediction (HiLP), which introduces an auxiliary higher-level abstract latent to help reduce the error accumulation effect in latent-space rollouts. Experiments show that HiLP can lead to longer-horizon coherent belief state representation and demonstrate the effectiveness of our method across coding and multi-step reasoning benchmarks, and offers more speculative decoding efficiency.
Changyan Shi, Tim Pearce, Manan Tomar et al.· 0 citations
The design and launch of SnapLGR, an LLM-based generative retrieval system for short-video recommendation at Snapchat shows that successful production SnapLGR requires joint design across representation learning, vocabulary grounding, and efficient training and serving.
Liam Collins, Jiwen Ren, Donald Loveland et al.· 0 citations