Skip to content
Open access

A reinforcement learning-driven framework for real-time human-computer interaction: latency-aware composite reward optimization based on PPO

Oct 2026 · Journal of engineering and applied sciences · Vol 73 · 0 citations · 28 references

TL;DR

The proposed latency-aware composite reward optimization framework combines informative sequence-level evaluation with step-wise policy regularization to improve empirical training stability and demonstrate the potential of latency-aware composite reward optimization to balance response quality, lexical diversity, and inference efficiency in real-time domain-specific LLM applications.

Abstract

While Large Language Models (LLMs) exhibit remarkable zero-shot learning capabilities, their deployment in real-time interactive systems is constrained by sparse terminal feedback in long-sequence generation, unstable policy updates, and the computational overhead associated with verbose outputs. To address these challenges, this paper formalizes autoregressive text generation as a discrete-time Markov decision process (MDP) and proposes a latency-aware composite reward optimization framework based on proximal policy optimization (PPO). The core contribution of the proposed method is a multidimensional sequence-level terminal reward that jointly evaluates semantic coherence, lexical diversity, and computational cost. Specifically, a Natural Language Inference (NLI) cross-encoder evaluates the semantic consistency between the input context and the complete generated response, while Shannon entropy and Distinct-N metrics encourage lexical diversity. In addition, a hardware-aware penalty based on generated sequence length and measured inference latency discourages unnecessarily verbose outputs and promotes computationally efficient generation. During policy optimization, the terminal composite reward is propagated through generalized advantage estimation, while a token-wise Kullback–Leibler divergence term regularizes the updated policy against the reference policy. This formulation does not introduce token-level semantic rewards; instead, it combines informative sequence-level evaluation with step-wise policy regularization to improve empirical training stability. We evaluated the proposed framework on a large-scale hybrid corpus containing approximately 320,000 dialogue turns, including real-world English interaction data. Compared with the standard PPO baseline, the proposed method exhibited faster empirical convergence and lower variability in the observed training trajectories. It achieved a BLEU-4 score of 22.87 and a Distinct-2 score of 21.59%, while reducing the average inference latency to 42.8 milliseconds per token under the reported hardware configuration. These results demonstrate the potential of latency-aware composite reward optimization to balance response quality, lexical diversity, and inference efficiency in real-time domain-specific LLM applications.

Read PDF

Similar papers

Using Context Is Not Enough: Test-Time Training for Personalized Reward Modeling

Preference-Aligned Test-Time Training (P-TTT) is proposed, which explicitly encodes preference relations conveyed by contextual pairs into user-specific fast weights for personalized reward prediction and outperforms state-of-the-art methods by a large margin.

Bo-Hao Wang, Xiao-Yan Zhao, Yang Zhang et al. · 0 citations
Open access 2026

Dual-Group Exploration: Reinforcement Learning for Video Understanding in Multimodal Large Language Models

Reinforcement Learning (RL) has significantly enhanced reasoning in Large Language Models. However, applying state-of-the-art RL methods like Group Relative Policy Optimization (GRPO) to video-based Multimodal LLMs (MLLMs) remains sub-optimal. We identify two key exploration bottlenecks: (1) Advantage Vanishing : gener...

Qu-Cheng Peng, Rui-Yun Hou, Jing-Yi Zhang et al. · 0 citations
#artificial intelligence Preprint Oct 2026

RewardWeaver: Long-Horizon Interactive Learning for Language Agents via Self-Evolving Reward Adaptation

RewardWeaver is introduced, a self-evolving reward adaptation framework for language agents in long-horizon interaction that maintains a validated capability space in which the semantics of admitted Rubrics remain fixed, and closes the loop between policy optimization, task evaluation, failure attribution, and reward a...

Heng-Bo Xiao, Bo-Yao Zhang, Pu-Rui Liu et al. · 0 citations

TTRSD: Test-Time Reinforcement Learning with Self-Distillation for Vision-Language Models

TTRSD separates update direction from update position, determined by group-relative advantages, from update position, determined by visual sensitivity, without requiring ground-truth labels, external verifiers, or a separate teacher, demonstrating cross-dataset generalization while preserving inherent reasoning integri...

Shu-Ning Wang, Zhi-Heng Wu, Xun-Lan Zhou et al. · 2 citations
Review Open access Oct 2026

Reinforcement Learning: A Comprehensive Review of Foundations, Algorithms, and Open Challenges

Reinforcement learning (RL) is a branch of machine learning that studies how an agent should act within an environment to maximize cumulative reward. Supervised learning fits a function to labeled examples; RL instead learns through the consequences of trial-and-error interaction. It also differs from unsupervised lear...

Mohammad Hatim Eliwi, Waleed A. Hammod, O. A. Hammood et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Variance-Averse $n$-Step Offline Reinforcement Learning for Sparse Long-Horizon Environments

Generative actors are transforming offline reinforcement learning (RL) by enabling expressive policy classes that model complex action distributions. However, this expressiveness also exposes a key challenge in heterogeneous datasets: generative policies can reproduce unreliable action modes whose return distributions...

Gu-Heon Kang, Minhae Kwon · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.