Jun 2026· IEEE Conference on Network Softwarization· pp. 207-212· 0 citations· 13 references
Computer Science
Abstract
A key pillar of the evolving telco cloud landscape is the potential hosting of AI workloads such as Large Language Model (LLM) inferencing instances. While Graphics Processing Units (GPUs) are often favoured for best possible performance, it is also feasible to run certain LLM models on more standard Central Processing Unit (CPU) based compute infrastructure. Indeed, multiple LLMs can run in a “multi-tenant” format, as long as suitable resource isolation mechanisms (particularly CPU core pinning/affinity) is enforced. Within an edge compute environment where resources are generally more constrained, including the ready availability of power, it is crucial to achieve a balance between attainable “good enough” performance of LLMs, while also conserving compute power. A tunable entity of many recent families of Intel x86 servers is the “uncore” component of the processor socket. The uncore functions are separate from actual processor cores including last level cache, memory and I/O control systems. This paper explores how detuning uncore frequency can modulate LLM performance by reducing it deterministically to within acceptable bounds. Since a corresponding power reduction is achievable, this is a very promising approach to achieving LLM energy efficiency at the network edge. As an example, for three concurrent llama3 instances, reducing uncore frequency from 2200 MHz to 1500 MHz drops the Tokens Per Second (TPS) from 6.78 to 6.45 with a corresponding 12 W power saving; further de-tuning to 1100 MHz yields a 32 W power saving while reducing TPS to 5.74, a value that is still marginally above a target “minimum acceptable” threshold.
OpScale is presented, a practical operator-level orchestration framework of profiling, provisioning, placement, and runtime serving that attains SLOs with up to 36.3% fewer GPUs and 28% less power, or achieves 44% higher throughput under fixed cost budgets.
Xingqi Cui, Chieh-Jan Mike Liang, Ziang Tang et al.· 0 citations
The main result concerns the plain-load path: attained LDG bandwidth peaks at a small offered per-thread load (K ~ 2) and then declines, by about 35% from K=2 to K=8 at the authors' primary configuration.
Frontier open-weight models are increasingly available, but serving them still largely assumes datacenter infrastructure. We present FreeToken, an edge-native MoE serving system that treats a personal machine not as a small GPU, but as a unified, elastic inference platform. FreeToken co-designs the full serving stack, including model layout and loading, expert residency, CPU--GPU execution, agentic state reuse, and runtime memory management, around two realities of local AI: agent workloads continuously change their execution pattern, and edge hardware exposes heterogeneous resources whose balance differs from machine to machine. Rather than committing to a fixed offloading strategy, FreeToken continuously maps computation and model state onto the resources actually available. FreeToken supports more than 20 MoE models and real coding and tool-using agents across hardware ranging from an 8GB laptop GPU to a single workstation GPU. More importantly, it changes what these machines can practically serve, from a 35B model on a laptop to a 284B model on a gaming desktop and the 753B GLM-5.2 on a single workstation GPU. FreeToken turns open weights into deployable local software, making the machines users already own a practical platform for frontier-scale intelligence. We release the system at flashml.ai.
Shuo Yang, Xiao-yun Fan, Melissa Z. Pan et al.· 2 citations
FPGA-based SmartNICs are widely deployed for cloud network function acceleration, but their memory subsystem is under increasing pressure because of large session tables. Conventional wisdom suggests that high packet processing performance relies on advanced memories (e.g., SRAM, HBM), but those are costly to add at cloud scale. Dorado is a novel design that scales SmartNIC session tables entirely on inexpensive DDR modules. At the heart of Dorado are three new techniques that extract commodity DDR performance by restructuring session table layout, decomposing processing pipelines to reduce locking, and scheduling memory accesses to minimize stalls. Our testbed results show that Dorado improves packet processing rates by 33%, even with fewer hardware resources. Further, we have deployed Dorado to millions of servers, processing network traffic from billions of users on a large public cloud for over three years. Our production results show that Dorado can accommodate up to 16M session entries, reduce memory cost by 80%, while enabling 50Mpps line-rate processing.
Heng Yu, Kai Ren, Jiajun Liang et al.· Proceedings of the ACM SIGCO...· 0 citations