InferenceDynamics: Adaptive LLM Routing through Structured Capability and Knowledge Profiling

WISERouter: LLM Routing with Workload Budget Constraint

Empirical results on RouterBench and SWE-Bench demonstrate that WR-Offline surpasses existing baselines in performance under a fixed budget and adheres more closely to budget constraints, and WR-Online achieves comparable performance to the baselines, while using substantially less exploration data.

LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers

This work presents a unified formulation of LLM routing as a sequential decision process characterized by five components: context encoders, model encoders, scoring functions, decision rules, and learning signals, covering single-turn, multi-turn, and personalized routing.

LLM-Advisor: Dynamic Model Selection and Query Routing in Heterogeneous Multi-LLM Architectures

LLM-Advisor is introduced, an open-source, adaptive framework designed for intelligent query categorization, dynamic model evaluation, and constraint-aware request routing across heterogeneous multi-LLM pools that achieves a 42% reduction in overall inference expenditure and a 35% decrease in average response latency.

Adaptive Cascading and Gated Distillation for Cost-Efficient Multi-Model Collaboration

Adaptive Ranks for Personalized Federated Large Language Models Under Parameter Budget Constraints

The advent of Large Language Models (LLMs) represents a leap toward leveraging AI for the benefit of humanity. However, the realization of this potential requires addressing the growing need for personalization and data privacy across diverse entities. Federated Learning (FL) offers a vital paradigm for developing private and personalized LLMs, yet its effectiveness is often limited by the inequality of computational resources at the edge. Existing federated Low-Rank Adaptation (LoRA) struggles to accommodate such heterogeneous hardware capacities, leading to information loss during model aggregation. To bridge this gap, this paper presents an adaptive aggregation framework designed to optimize federated LLM fine-tuning under these constrained conditions. The framework introduces a dynamic budgeting mechanism that quantifies personalization intensity through Frobenius norm divergence and allocates rank capacity accordingly. To resolve rank heterogeneity, a dimension-aligned strategy based on Singular Value Decomposition (SVD) is applied, enabling the consistent fusion of updates across diverse devices. Extensive experiments on the General Language Understanding Evaluation (GLUE) benchmark using Llama-3-8B achieve an average score of 0.6551 and a mean task ranking of 2.22. The results indicate that our method improves global generalization while preserving client-specific personalization in resource-limited environments.

Dynamo-MoE: Accelerating Sparse Large Model Inference with Dynamic Parallelization

Mixtral-of-Experts (MoE) has become one of the major model structures in LLMs because of its computational efficiency when scaling the model size. However, MoE model inference suffers from critical load imbalance issue caused by the sparsely and dynamically activated experts. In addition, current inference frameworks are oblivious to the real-time workload fluctuation, a common phenomenon in LLM serving. Therefore, the static model deployment of existing frameworks leads to severe performance limitations. To this end, we propose Dynamo-MoE, an out-of-box MoE inference framework to bridge the performance gap by dynamic parallelization strategies. Specifically, Dynamo-MoE integrates a novel load balancing approach based on token sorting and on-demand expert loading to solve the workload imbalance issue in the scenario of high workload (such as Prefill). Dynamo-MoE is also aware of workload varying to adaptively switch between tensor parallelism (for low latency in small batch scenarios) and expert parallelism (for high throughput in large batch scenarios). Furthermore, the model parameter redistribution overhead of dynamic parallelization is smartly overlapped through sophisticated pipeline orchestration. Compared to the SOTA framework vLLM (w/ and w/o EPLB), Dynamo-MoE achieves up to 6.75 × reduction for TTFT, 1.59 × reduction for TPOT, and 1.5 × improvement for throughput.