This survey aims to systematically address resource efficiency challenges in Large Language Models by reviewing a broad spectrum of techniques designed to enhance the resource efficiency of LLMs, and introduces a nuanced categorization of resource efficiency techniques by their specific resource types.
Abstract
The burgeoning field of Large Language Models (LLMs), exemplified by sophisticated models like OpenAI’s ChatGPT, represents a significant advancement in artificial intelligence. These models, however, bring forth substantial challenges in high consumption of computational, memory, energy, and financial resources, especially in environments with limited resource capabilities. This survey aims to systematically address these challenges by reviewing a broad spectrum of techniques designed to enhance the resource efficiency of LLMs. We categorize methods based on their optimization focus—covering computational, memory, energy, financial, and network resources—and their applicability across various stages of an LLM’s lifecycle, including architecture design, pre-training, fine-tuning, and system design. Additionally, the survey introduces a nuanced categorization of resource efficiency techniques by their specific resource types, which uncovers the intricate relationships and mappings between various resources and corresponding optimization techniques. A standardized set of evaluation metrics and datasets is also presented to facilitate consistent and fair comparisons across different models and techniques. By offering a comprehensive overview of the current state-of-the-art and identifying open research avenues, this survey serves as a foundational reference for researchers and practitioners, aiding them in developing more sustainable and efficient LLMs in a rapidly evolving landscape. To make the field more accessible to interested beginners, we not only make a systematic review of existing works and a highly structured taxonomy of resource-efficient LLMs but also release a website including a constantly-updated paper list https://github.com/tiingweii-shii/Awesome-Resource-Efficient-LLM-Papers.
This comprehensive survey provides an in-depth analysis of modern LLMs, with a systematic comparison of cutting-edge proprietary and open-source architectures including DeepSeek, GPT-4, Gemini, LLaMA, and Claude.
This work evaluates 70+ SLMs from 2023–2025 on five task-specific benchmarks and compares them with two popular LLMs, revealing key trade-offs between energy, performance, and model selection and highlighting the need for informed, task-aware model selection rather than size-driven choices.
Fabian Reichwald, Lukas Schiesser, Christiane Plociennik et al.· Proceedings of the Thirty-Fi...· 0 citations
This survey examines how these methods integrate graphs into various stages of the LLM pipeline, including the input, model, and output phases, and outlines the challenges and future research directions for developing more efficient and interpretable solutions.
Xin-Yan Zhu, Cheng Yang, Qiu-Yue Wang et al.· Proceedings of the Thirty-Fi...· 0 citations
This tutorial provides a comprehensive and up-to-date overview of LLM mechanism discovery, validation, and editing, and surveys mechanistic editing techniques that leverage MI insights to modify behavior at varying granularity.
Yin-Han He, Wendy Zheng, Tianyi Zhao et al.· Proceedings of the 32nd ACM...· 0 citations
This survey reviews the evolution of language models from early statistical approaches to modern Transformer-based architectures and summarizes key developments, including attention mechanisms, scaling laws, alignment techniques, and efficient inference methods.
P. Peykani, V. Charles, A. Emrouznejad et al.· Archives of Computational Me...· 0 citations
This work systematically map 14,767 papers introducing or updating evaluation resources from arXiv submissions between January 2022 and August 2026, using staged screening and automated full-text coding to examine changes in target systems and domains, evaluation materials and conditions, and scoring mechanisms.
Chao Wang· arXiv.org· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.