A Cloud--Edge Collaborative Large Language Model Inference Framework Based on Historical Context Matching

Abstract

Cloud–edge collaborative inference has emerged as a promising paradigm to address the latency, energy, and privacy challenges of large language models (LLMs). However, current offloading mechanisms often struggle to efficiently capture the dynamic semantic dependencies between historical context and ongoing queries. This oversight results in redundant token processing or imprecise cloud routing, leading to semantic disconnection and suboptimal system utility. To overcome these limitations, we propose a cloud–edge collaborative LLM inference framework with historical context matching (HCM-CELLM), which integrates context awareness with edge computing. Deployed at the edge, HCM-CELLM employs an approximate nearest neighbor (ANN) search and a lightweight Transformer-based attention module to assess semantic relevance, filtering ambiguity and injecting only high-value historical segments. The dynamic matching degree serves as a core state feature for an improved proximal policy optimization (PPO) scheduler, enhanced with a trust-region caching mechanism to enable precision-aware offloading decisions—intercepting highly matched queries at the edge or coordinating cloud-sketch parallel refinement. Extensive simulations demonstrate that HCM-CELLM significantly outperforms state-of-the-art collaborative baselines, achieving superior inference accuracy, task completion rates, and a reduction in latency and energy consumption.