Skip to content

Replacing Large Language Models with Jev Decision Models for Low-Latency Edge Service Orchestration

Sep 2026 · arXiv.org · Vol abs/2609.22753 · 12 citations · ⚡ 3 influential · 46 references
Computer Science

TL;DR

Jev's decision-oriented application programming interface (API) is integrated into edge service orchestration to reduce overhead while retaining service completion, and decision-model substitution for latency-bound admission on bounded contracts is supported.

Abstract

Natural-language service requests can require a language-model decision before execution starts, consuming part of the request's latency budget. We integrate Jev's decision-oriented application programming interface (API) into edge service orchestration to reduce this overhead while retaining service completion. The integration extracts four to eight bounded intent fields and applies a shared validator, admission policy, and scheduler, accounting for decision waiting throughout the request timeline. We compare Jev, two self-hosted decision models, and three hosted large language models (LLMs) on 8,280 verified requests and on a live admission path with modeled execution and a real optical character recognition service. Across 33 test conditions, Jev reduces median decision latency by 22.7-64.5% relative to the fastest LLM. This latency barely moves with input size, contract width, or catalog size. On four-field contracts, Jev's API fees per correct decision are 59.7-80.9% lower at a cost of a few exact-match points, while wide contracts mark the limit of the substitution. Receiving the service catalog with each request, Jev names unseen services as accurately as known ones. On the live admission path, Jev keeps 0.91-0.95 of requests exact and on time at loads where the LLMs fall below 0.1. Since caching repeated descriptions gives the interpreters nearly the same latency, Jev's gain lies in fresh decisions. These results support decision-model substitution for latency-bound admission on bounded contracts.

View source

Similar papers

Preprint Sep 2026

Intent Interpretation at RIC Timescales: Jev Decision Models versus Large Language Models in 6G Open RAN

This work compares Jev-1.13.13.0 and two other decision models with LLMs on the RANIntent v1 benchmark, in closed-loop ns-3 simulation and on a real A1 and E2 path, and asks whether the extra delay of LLMs costs control deadlines, RIC capacity, or radio performance.

De-Long Li, Xu Wang, Hao-Chen Gong et al. · 6 citations · ⚡1

StreamDecisionBench: Evaluating Decisions in Force on Evolving Language Streams

Language models increasingly make real-time decisions in applications that apply the latest answer until a newer one arrives. A late answer can prolong an outdated decision, such as a call recorder still running while a customer reads out card details, an error offline accuracy misses. We make three contributions. Firs...

Jhen-Ke Lin, Chung-Chun Wang · 0 citations
#edge computing Preprint Aug 2026

PRISM: Predictive Runtime In-place Scaling and Model Selection for Edge Microservices

PRISM, a prediction-guided runtime framework that jointly selects model variants and CPU allocations for containerized edge microservices, and adapts each pipeline stage in place and minimizes predicted CPU-package energy under deadline, resource, and offline model-level Quality of Result constraints is presented.

Uwe Gropengießer, Thomas Reuter, Dominik Schön et al. · 0 citations
#edge computing Preprint Sep 2026

Unified AI Gateway: A Framework for Joint Model Routing and KV Cache Management

This work defines and analyzes the Unified AI Gateway as a system setting for an edge-deployed AI traffic hub that coordinates model routing, KV cache management, and compute placement across end devices, edge resources, and cloud model services.

Jia-Xun Lu, Xiang Zhang, Yun-Feng Shao · 1 citation
#artificial intelligence Preprint Sep 2026

Intent2Tc: Automated Intent-to-Traffic Control Translation with Language Models

Intent2Tc is presented, a closed-loop language-model-driven framework that translates business-level traffic-shaping intents into declarative sub-intents and subsequently into validated, executable Linux traffic control (tc) configurations and demonstrates the practical applicability of the proposed framework.

Andrea Masini, Sudipta Acharya, P. Bellavista et al. · 1 citation
#artificial intelligence Preprint Oct 2026

JOVE: Joint Execution and Verification for Resource-Aware LLM Task Graphs

Complex reasoning queries can be decomposed into directed acyclic task graphs and distributed across heterogeneous LLMs, reducing latency through parallelism and enabling smaller models to solve complex tasks. In practice, however, the suitability of an LLM for a given subtask may be a priori unknown, and execution alo...

Hao-Ran Zhang, Dongjun Kim, Seohyeon Cha et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.