Oct 2026· Journal of engineering and applied sciences· Vol 73· 0 citations· 28 references
TL;DR
The proposed latency-aware composite reward optimization framework combines informative sequence-level evaluation with step-wise policy regularization to improve empirical training stability and demonstrate the potential of latency-aware composite reward optimization to balance response quality, lexical diversity, and inference efficiency in real-time domain-specific LLM applications.
Abstract
While Large Language Models (LLMs) exhibit remarkable zero-shot learning capabilities, their deployment in real-time interactive systems is constrained by sparse terminal feedback in long-sequence generation, unstable policy updates, and the computational overhead associated with verbose outputs. To address these challenges, this paper formalizes autoregressive text generation as a discrete-time Markov decision process (MDP) and proposes a latency-aware composite reward optimization framework based on proximal policy optimization (PPO). The core contribution of the proposed method is a multidimensional sequence-level terminal reward that jointly evaluates semantic coherence, lexical diversity, and computational cost. Specifically, a Natural Language Inference (NLI) cross-encoder evaluates the semantic consistency between the input context and the complete generated response, while Shannon entropy and Distinct-N metrics encourage lexical diversity. In addition, a hardware-aware penalty based on generated sequence length and measured inference latency discourages unnecessarily verbose outputs and promotes computationally efficient generation. During policy optimization, the terminal composite reward is propagated through generalized advantage estimation, while a token-wise Kullback–Leibler divergence term regularizes the updated policy against the reference policy. This formulation does not introduce token-level semantic rewards; instead, it combines informative sequence-level evaluation with step-wise policy regularization to improve empirical training stability. We evaluated the proposed framework on a large-scale hybrid corpus containing approximately 320,000 dialogue turns, including real-world English interaction data. Compared with the standard PPO baseline, the proposed method exhibited faster empirical convergence and lower variability in the observed training trajectories. It achieved a BLEU-4 score of 22.87 and a Distinct-2 score of 21.59%, while reducing the average inference latency to 42.8 milliseconds per token under the reported hardware configuration. These results demonstrate the potential of latency-aware composite reward optimization to balance response quality, lexical diversity, and inference efficiency in real-time domain-specific LLM applications.
Preference-Aligned Test-Time Training (P-TTT) is proposed, which explicitly encodes preference relations conveyed by contextual pairs into user-specific fast weights for personalized reward prediction and outperforms state-of-the-art methods by a large margin.
Bo-Hao Wang, Xiao-Yan Zhao, Yang Zhang et al.· arXiv.org· 0 citations
Reinforcement Learning (RL) has significantly enhanced reasoning in Large Language Models. However, applying state-of-the-art RL methods like Group Relative Policy Optimization (GRPO) to video-based Multimodal LLMs (MLLMs) remains sub-optimal. We identify two key exploration bottlenecks: (1) Advantage Vanishing : gener...
Qu-Cheng Peng, Rui-Yun Hou, Jing-Yi Zhang et al.· Conference on Empirical Meth...· 0 citations
RewardWeaver is introduced, a self-evolving reward adaptation framework for language agents in long-horizon interaction that maintains a validated capability space in which the semantics of admitted Rubrics remain fixed, and closes the loop between policy optimization, task evaluation, failure attribution, and reward a...
Heng-Bo Xiao, Bo-Yao Zhang, Pu-Rui Liu et al.· 0 citations
TTRSD separates update direction from update position, determined by group-relative advantages, from update position, determined by visual sensitivity, without requiring ground-truth labels, external verifiers, or a separate teacher, demonstrating cross-dataset generalization while preserving inherent reasoning integri...
Reinforcement learning (RL) is a branch of machine learning that studies how an agent should act within an environment to maximize cumulative reward. Supervised learning fits a function to labeled examples; RL instead learns through the consequences of trial-and-error interaction. It also differs from unsupervised lear...
Mohammad Hatim Eliwi, Waleed A. Hammod, O. A. Hammood et al.· EDRAAK· 0 citations
Generative actors are transforming offline reinforcement learning (RL) by enabling expressive policy classes that model complex action distributions. However, this expressiveness also exposes a key challenge in heterogeneous datasets: generative policies can reproduce unreliable action modes whose return distributions...
Gu-Heon Kang, Minhae Kwon· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.