Aug 2026· Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2· 0 citations· 7 references
TL;DR
This tutorial provides a comprehensive and up-to-date overview of LLM mechanism discovery, validation, and editing, and surveys mechanistic editing techniques that leverage MI insights to modify behavior at varying granularity.
Abstract
Large language models (LLMs) have demonstrated remarkable capabilities across a wide range of tasks, yet their internal mechanisms remain largely opaque, making it difficult to understand, predict, or control their behavior. As LLMs are increasingly deployed in high-stakes settings, this lack of transparency raises serious concerns about reliability and safety. Mechanistic interpretability (MI) has emerged as a promising approach to address this challenge, seeking to reverse-engineer the internal computations of LLMs into human-understandable mechanisms, i.e., an approximate high-level algorithm that the LLM implements with a subset of its components (a circuit) to complete a certain language task or exhibit a certain behavior. This tutorial provides a comprehensive and up-to-date overview of LLM mechanism discovery, validation, and editing. We begin by introducing foundational concepts, including features, components, computational graphs, and circuits, along with key notation. We then examine mechanism discovery through four methodological families: causal mediation, attribution, sparse decomposition, and optimization-based approaches. Next, we turn to mechanism validation, covering methods for verifying proposed mechanisms and emerging standards for rigorous evaluation. Building on these foundations, we survey mechanistic editing techniques that leverage MI insights to modify behavior at varying granularity, from fine-grained representation-level steering to coarser circuit-level interventions. Lastly, we outline open challenges and future research directions, including scalability of interpretability methods, evaluation benchmarks for mechanistic circuits, and the integration of interpretability with training-time objectives, aiming to inspire continued progress in understanding and governing large language models.
This tutorial provides a comprehensive, end-to-end view of LLM interpretability, transitioning from microscopic neural analysis to macroscopic application and deployment, and explores how these interpretability paradigms scale and inspire the design of frontier architectures, agentic systems, and thinking models.
Wei Zhang, Zhengfu He, Lucia Zhang et al.· Proceedings of the 32nd ACM...· 0 citations
Large Language Models (LLMs) are predominantly assessed based on their common sense reasoning, language comprehension, and logical reasoning abilities. While models trained in specialized domains like mathematics or coding have demonstrated remarkable advancements in logical reasoning, there remains a significant gap in evaluating their code generation capabilities. Existing benchmark datasets fall short in pinpointing specific strengths and weaknesses, impeding targeted enhancements in models’ reasoning abilities to synthesize code.
To bridge this gap, this thesis introduces two novel contributions: CodeEval and CodeQual. CodeEval is an innovative, pedagogical benchmarking method that mirrors the evaluation processes encountered in academic programming courses. It comprises a multi-dimensional benchmark dataset of 602 hand-crafted problems designed to rigorously evaluate LLMs across 24 distinct aspects of Python programming, covering three proficiency levels—beginner, intermediate, and advanced—and includes both class-based and function-based problem types with detailed problem specifications and comprehensive test suites achieving 99.1% coverage. To facilitate widespread adoption, we developed RunCodeEval, an open-source execution framework that provides researchers with a ready-to-use evaluation pipeline. Our evaluation of 15 state-of-the-art LLMs revealed consistent performance degradation with increasing complexity (validated statistically, Cohen’s d = 0.790) and universal struggles with advanced concepts like concurrency.
Code quality is inherently subjective, encompassing dimensions like readability, efficiency, and adherence to language idioms that traditional static metrics fail to capture adequately. While large language models can assess these subjective qualities, lightweight models offer practical advantages: seamless CI/CD pipeline integration, lower operational costs, and full control over model behavior. We investigate whether such models can learn to assess code quality by training on synthetic LLM annotations. We introduce CodeQual, a dataset of 5,819 code samples derived from five established sources spanning diverse domains—competitive programming, pedagogical problems, software engineering, and general benchmarks—scored by LLMs across five quality dimensions, with 655 human-annotated samples for evaluation. Our fine-tuned model, CodeQualBERT, not only matches LLM performance but exceeds inter-human agreement on all five dimensions, achieving 16–100% improvement over the inter-human agreement baseline.
Together, these contributions provide a comprehensive framework for evaluating and improving LLMs in software engineering contexts, encompassing both functional correctness assessment and subjective code quality evaluation.
This systematic literature review (SLR) provides a comprehensive overview of pruning techniques applied to LLMs, based on 60 peer-reviewed studies and preprints published between 2022 and 2025, sourced from major digital libraries.
F. Bazikar, Atefeh Hemmati, Akram Reza et al.· Knowledge and Information Sy...· 0 citations
This paper identifies patch verbosity as a major yet overlooked concern in LLM-based APR and proposes RECAP, a lightweight, plug-and-play adapter that attaches to existing repair frameworks after generation that achieves a substantially better size-correctness tradeoff.
Wenqiang Luo, J. Keung, Xiaoyu Shi et al.· 0 citations
