Skip to content

Do LLMs Understand Context? A Knowledge Graph-Based Evaluation Framework

Sep 2026 · 0 citations · 34 references
Computer Science

TL;DR

A novel knowledge graph (KG) based evaluation framework for LLM contextual understanding in QA is proposed and central to this is Semantic Structural Similarity for KGs (S3KG), a hybrid similarity measure combining structural and semantic signals into a single score.

Abstract

While large language models (LLMs) have achieved remarkable linguistic capabilities, a profound question lingers at their core: do these models truly comprehend context or simply excel at pattern matching on an unprecedented scale? Contextual understanding in LLMs refers to the ability to correctly extract relevant information from a given context, integrate it into a coherent internal representation, and reason over it to produce factually consistent and contextually grounded responses. However, traditional methods such as BiLingual Evaluation Understudy (BLEU) and perplexity simply measure surface-level performance. This reveals a critical gap in question answering (QA), where responses must be contextually grounded rather than simply being memorized associations. To fill this void, we propose a novel knowledge graph (KG) based evaluation framework for LLM contextual understanding in QA. Central to this is Semantic Structural Similarity for KGs (S3KG), a hybrid similarity measure combining structural and semantic signals into a single score. In addition, a diagnostic analysis framework is developed to identify and categorize reasoning errors at the triplet level, enabling fine-grained analysis of model failures. Together, across nine benchmarks, S3KG achieves F1 gains of up to $+7.6$ points over the strongest baseline and AUROC up to $0.973$.

View source

Similar papers

#machine learning Preprint Sep 2026

Evaluation of Contextual Understanding in Large Language Models

A novel knowledge graph-based evaluation framework is proposed introducing Semantic Structural Similarity for KGs (S3KG), a hybrid similarity measure integrating structural and semantic similarity into a continuous evaluation score, alongside a diagnostic framework for categorizing reasoning errors.

Subavarshana Arumugam, Mamta Nallaretnam, K. Wickramasinghe et al. · 0 citations
Open access 2026

Semantic Parsing for Evaluating Large Language Models: Separating Linguistic Abilities with YARN

A layer-wise analysis indicates that surface-level features such as temporality and negation are captured more reliably than deeper semantic phenomena like quantification in large language models, highlighting the limited capacity of current LLMs to generate fully formal meaning representations.

Rémi De Vergnette, Maxime Amblard · 0 citations
#artificial intelligence Open access Sep 2026

Can We Do Interpretable NLI with Graphs Based on Atomic Propositions?

This paper introduces a fully graph-based pipeline where the classifier never directly processes the input text, and reveals that graphs and text are complementary: combining both modalities achieves 92.1% accuracy on SNLI.

Younes Boufouss, Luc Pommeret, Thomas Gerald et al. · 0 citations
#artificial intelligence Preprint Sep 2026

DocHop: Benchmarking Out-of-domain Multi-hop Reasoning in Information-Dense Documents

This work introduces DocHop, a benchmark for integrated chart--context reasoning in document-style images and constructs DocHop via a stochastic logic-first generation pipeline with controllable reasoning depth and visual density, to enable systematic evaluation.

Zhuo-Ran Yu, Le Thien Phuc Nguyen, Jaden Park et al. · 1 citation
#artificial intelligence Preprint Aug 2026

Do General NLP Embeddings Capture Ontological Reasoning?

AVA is introduced, a systematic framework for evaluating whether embeddings distinguish logic-sensitive relational semantics in ontologies and knowledge graphs, and reveals a persistent gap between linguistic representation learning and ontology-level discrimination, challenging the assumption that strong NLP benchmark...

Hamed Babaei Giglou, Jennifer D'Souza, S. Auer · 0 citations
Open access 2026

Integrating Knowledge Graph with Large Language Models for Multi-hop Question Generation

This work proposes Knowledge Graph for Question Generation (KG4QG), a novel framework that integrates knowledge graphs with large language models to address the challenges of MHQG, and demonstrates the effectiveness of combining structured knowledge and pre-trained language models for complex question generation tasks.

Yllias Chali, Al Hasib Mahamud · 0 citations

Related blog posts

MIT News · Artificial Intelligence Sep 29, 2026

Who we become when we talk to machines

Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.