Dec 2025· arXiv.org· Vol abs/2512.05525· 1 citation· 45 references
Computer Science
TL;DR
This paper presents the vision for just-in-time model replacement (JITR), where, upon identifying a recurring task in calls to an LLM, the model is replaced transparently with a cheaper alternative that performs well for this specific task.
Abstract
Businesses increasingly rely on large language models (LLMs) to automate simple repetitive tasks instead of developing custom machine learning models. LLMs require few, if any, training examples and can be utilized by users without expertise in model development. However, this comes at the cost of substantially higher resource and energy consumption compared to smaller models, which often achieve similar predictive performance for simple tasks. In this paper, we present our vision for just-in-time model replacement (JITR), where, upon identifying a recurring task in calls to an LLM, the model is replaced transparently with a cheaper alternative that performs well for this specific task. JITR retains the ease of use and low development effort of LLMs, while saving significant cost and energy. We discuss the main challenges in realizing our vision regarding the identification of recurring tasks and the creation of a custom model. Specifically, we argue that model search and transfer learning will play a crucial role in JITR to efficiently identify and fine-tune models for a recurring task. Using our JITR prototype Poodle, we reduce inference time by up to 7.5x compared to a self- hosted LLM and save more than $2,200 per 1M requests compared to a flagship hosted LLM, while achieving accuracy competitive with the LLM baseline.
A detailed classification of LLMs is showcased, starting with the earlier models like BERT and leading to LLMs like GPT, Gemini, Claude and Grok, revealing a clear pattern: models that optimize the ratio between architectural complexity and data quality exhibit stronger reasoning stability and better alignment, without...
In recent years, the performance of large language models (LLMs) on reasoning tasks has been remarkable, even surpassing human capabilities on various benchmarks. However, there remains a lack of clear understanding in the academic community regarding how the structure and internal parameters of LLMs progressively solv...
A systematic, multi-model framework that combines Chain-of-Thought prompting with Low-Rank Adaptation parameter-efficient fine-tuning, and measures the contribution of each component independently is investigated, making capable SLM-based reasoning deployable without datacenter-scale infrastructure.
Aryan Raina, Shiwani Gupta, Jagruti Jadhav et al.· Journal of Computer Science· 0 citations
This survey aims to systematically address resource efficiency challenges of Large Language Models by reviewing a broad spectrum of techniques designed to enhance the resource efficiency of LLMs, and introduces a nuanced categorization of resource efficiency techniques by their specific resource types.
The results provide a case study in which a medically fine-tuned checkpoint does not improve jargon comprehension over its general-purpose counterpart, highlighting that domain adaptation should not be assumed to yield better performance on specialized terminology.
Darin Keng, Zhe-Wei Sun· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduNov 17, 2026
By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.
Microsoft Research Blog· microsoft.comNov 16, 2026
AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.
MIT News · Artificial Intelligence· news.mit.eduNov 13, 2026