This work bridges hybrid modeling and neuro-symbolic (NeSy) AI by reconstructing these designs as instances of NeSy interface and derives metrics: structural violation rate (SVR), measuring whether the learned belief respects the mechanistic structure; and belief dispersion (BD), measuring how concentrated the learned plausibility is, serving as a hybrid model's epistemic uncertainty in its mechanistic part.
Abstract
Hybrid mechanistic/data-driven models, which combine first-principles with learned components, are increasingly used in process engineering and scientific machine learning. Common hybrid modeling designs are specified primarily through their architectures and training losses, which offers a limited basis for a shared semantic interface to compare or verify them across domains, with comparatively little attention paid to epistemic uncertainty in the mechanistic part. We bridge hybrid modeling and neuro-symbolic (NeSy) AI by reconstructing these designs as instances of NeSy interface. The resulting translation, Hybrid-to-NeSy (H2N), places mechanistic knowledge on the language side, learned modules on the belief side, and validity domains together with constraints on the logic side. For each design, H2N then yields an explicit NeSy inference functional and a logic-belief decomposition. From this decomposition we derive two metrics: structural violation rate (SVR), measuring whether the learned belief respects the mechanistic structure; and belief dispersion (BD), measuring how concentrated the learned plausibility is, serving as a hybrid model's epistemic uncertainty in its mechanistic part. We instantiate H2N on a case study of a structured hybrid model for binary classification under label noise and show that models with higher SVR and BD exhibit greater variability in held-out accuracy. Under structural distribution shift, H2N further quantifies a model's uncertainty during extrapolations, whereas test accuracy reveals the same shift only post hoc.
This article offers a comprehensive overview of mechanistic interpretability, an emerging field that seeks to reverse-engineer the internal algorithms of modern neural networks. While traditional explainable AI methods often stop at surface-level input-output correlations, this approach directly addresses the opaque"black box"nature of machine learning models, which is essential for ensuring safety and auditability in high-stakes deployments. The paper provides a detailed examination of Transformer circuit analysis, exploring how internal components like the residual stream, attention mechanisms, and induction heads drive complex tasks and in-context learning. It subsequently tackles the core challenge of superposition and polysemanticity, demonstrating how tools like Sparse Autoencoders (SAEs) and transcoders can decompose tangled network activations into distinct, human-interpretable features. Furthermore, the paper explores methods for actively controlling and modifying model behavior through steering vectors and causal interventions. Finally, it connects these mechanistic insights with neurosymbolic AI frameworks designed to translate neural representations into explicit, executable logical rules.
Background: Deep neural networks increasingly power language, vision, and decision systems, yet many deployments require explanations that are faithful, compositional, and governance-ready. Symbolic techniques promise these properties, but the literature mixes post-hoc extraction, knowledge injection, and intrinsically hybrid designs without a unifying view.
Objectives: We provide a systematic review and synthesis of symbolic explainable AI (XAI) for deep learning (January 2017– June 2025), organize the field around a three-part taxonomy—Symbolic Knowledge Extraction (SKE), Symbolic Knowledge Injection (SKI), and Hybrid neurosymbolic architectures—and propose a conceptual framework that clarifies training–inference flows, explanation interfaces, human feedback, and governance touchpoints.
Methods: Beginning from ≈50,000 records, we deduplicated and screened full texts, analyzed 393 PDFs, and included 273 primary studies in the synthesis. We coded each paper for model domain, modality, symbolic formalism, explanation scope and stage, evaluation protocol, and governance alignment. Analyses combine descriptive statistics with stratification by domain and formalism; we qualitatively assess evidence for faithfulness, robustness, data efficiency, and constraint satisfaction.
Results: Research activity accelerates after 2020, with a marked turn toward hybrids. Across the corpus, SKE, SKI, and Hybrid account for approximately 29%, 26%, and 45% of studies, respectively. Rule sets/decision trees remain the dominant explanation artifacts, while logic- and program-based formalisms grow in NLP and planning. SKI most often targets constraint satisfaction and robustness improvements; SKE emphasizes global surrogates and faithfulness auditing; hybrids report gains in sample efficiency and traceable reasoning. However, evaluation practices are heterogeneous, human-subject studies are scarce, and explicit links to policy/risk controls appear in a minority of works.
Conclusions: Our framework unifies how data, priors, and symbolic reasoning interact with neural learners, the explanation interface, human stakeholders, and governance. We distill actionable recommendations: (1) report faithfulness and constraintsatisfaction metrics alongside accuracy; (2) specify symbolic assumptions and training-time injections precisely; (3) include user studies or auditor-centric protocols for high-stakes use; and (4) develop benchmarks that couple tasks with machinereadable knowledge bases. We highlight open problems in scalable formal reasoning with foundation models, verifying generated rationales, and measuring causal faithfulness at scale.
Eduard Ionel Stan, G. Sciavicco, Paolo Napoletano· Journal of Artificial Intell...· 0 citations
Neuro-symbolic (NeSy) AI aims to develop deep neural networks whose predictions comply with prior knowledge encoding, e.g., safety or structural constraints. As such, it represents one of the most promising avenues for reliable and trustworthy AI. The core idea behind NeSy AI is to combine neural and symbolic steps: neural networks are typically responsible for mapping low-level inputs into high-level symbolic concepts, while symbolic reasoning infers predictions compatible with the extracted concepts and the prior knowledge. Despite their promise, it was recently shown that – whenever the concepts are not supervised directly – NeSy models can be affected by Reasoning Shortcuts (RSs). That is, they can achieve high label accuracy by grounding the concepts incorrectly. RSs can compromise the interpretability of the model’s explanations, performance in out-of-distribution scenarios, and therefore reliability. At the same time, RSs are difficult to detect and prevent unless concept supervision is available, which is typically not the case. However, the literature on RSs is scattered, making it difficult for researchers and practitioners to understand and tackle this challenging problem. This overview addresses this issue by providing a gentle introduction to RSs, discussing their causes and consequences in intuitive terms. It also reviews and elucidates existing theoretical characterizations of this phenomenon. Finally, it details methods for dealing with RSs, including mitigation and awareness strategies, and maps their benefits and limitations. By reformulating advanced material in a digestible form, this overview aims to provide a unifying perspective on RSs to lower the bar to entry for tackling them. Ultimately, we hope this overview contributes to the development of reliable NeSy and trustworthy AI models.
E. Marconato, Samuele Bortolotti, Emile Van Krieken et al.· Journal of Artificial Intell...· 0 citations
This tutorial provides a comprehensive, end-to-end view of LLM interpretability, transitioning from microscopic neural analysis to macroscopic application and deployment, and explores how these interpretability paradigms scale and inspire the design of frontier architectures, agentic systems, and thinking models.
Wei Zhang, Zhengfu He, Lucia Zhang et al.· Proceedings of the 32nd ACM...· 0 citations
: The design and specification of experiments in Model-Based Systems Engineering is challenging: state-of-the-art tools are deemed either precise, but too cumbersome or too imprecise due to natural-language descriptions that lack formal semantics. This is compounded by the high complexity of systems, especially in safety-critical domains. Large Language Models (LLMs) offer a promising avenue for automating the elicitation step, but their probabilistic nature precludes unmediated use: hallucinations cannot be allowed to propagate into formal artifacts. We propose a neuro-symbolic framework combining LLM-driven elicitation constrained by a rule-based reasoner fed by an ontology-compliant knowledge graph. A deterministic orchestrator drives an elicitation loop where the symbolic engine poses context-sensitive questions, the LLM proposes candidate answers, and every candidate is validated against formal domain constraints before acceptance. We present a proof-of-concept implementing the proposed framework and an empirical evaluation across three case studies using four state-of-the-art LLMs. Results indicate that the framework reliably prevents hallucinations from propagating into formal specifications.
Diego Ferreira, Rakshit Mittal, Lucas Lima et al.· Proceedings of the 16th Inte...· 0 citations
It is argued that mechanistic interpretability has the potential to support a more scientific understanding of machine learning systems – treating models not only as tools for solving tasks, but also as systems to be studied and understood.