Oct 2026· Frontiers of Computer Science· Vol 8· 0 citations· 24 references
TL;DR
The study evaluates generation quality, intervention sensitivity, and cross-stage consistency in a large language model-based pipeline for structured processing of scientific abstracts and for generating paired factual and counterfactual question-answer (QA) instances under document-level intervention constraints.
Abstract
A large language model (LLM)-based pipeline is presented for structured processing of scientific abstracts and for generating paired factual and counterfactual question-answer (QA) instances under document-level intervention constraints.
The implemented system extracts entities, events, and relations; induces a lightweight structural causal model (SCM) over scientific variables such as method, data, intervention, outcome, and metric; generates five factual QA pairs per abstract; rewrites them under one selected intervention; and filters inconsistent counterfactuals. The source export contains 100 Scopus abstracts, of which 67 documents with complete pipeline artifacts form the validation cohort. Generation uses
gpt-4.1
, while automated validation uses
gpt-5-mini
under a 15-criterion checklist; three human annotators independently score the same 67 documents.
Human agreement is moderate, with a mean pairwise Cohen's kappa of approximately 0.54. Automated validation is high for original QA and relation-level correctness (0.97 and 0.89) but very low for document-level global consistency (0.07), revealing a substantial gap between local correctness and cross-stage coherence. Under the score-based intervention analysis, 25.4% of documents exhibit a validation decrease above the predefined threshold.
We refer to this quantity as the intervention degradation rate rather than as bias propagation, because the interventions concern scientific-content variables rather than protected attributes. The study evaluates generation quality, intervention sensitivity, and cross-stage consistency; it does not train a foundation model or claim training-time bias mitigation.
A schema-grounded, tiered evaluation framework for structured scientific evidence extraction that increases task difficulty from single-entity identification to relational tuples and corpus-level statistical tasks and provides a reusable query suite and diagnostic evaluation framework to support future fine-grained ben...
Zhiyin Tan, Jennifer D'Souza, Sören Auer· International Journal on Dig...· 0 citations
Abstract Background Large language models (LLMs) exhibit extensive medical knowledge but are prone to hallucinations and show low fact-level explainability, limiting clinical adoption and regulatory compliance. Existing approaches, such as retrieval-augmented generation, partially address these issues by grounding answ...
J. Vladika, A. Domres, Mai Q. Nguyen et al.· Journal of Medical Internet...· 0 citations
Most automated survey generation systems retrieve papers as flat documents, limiting their ability to capture research structure. We present SurveyGraph, a framework built around a Methods-Entity Knowledge Graph (MEKG) encoding method-level follow-on, competition, and shared-resource relations. The graph supports core-...
Yi-Fan Yuan, Yan-Cheng Zhu, Zuo-Yu Zhang et al.· Conference on Empirical Meth...· 0 citations
EviGraphRAG is proposed, a ranker-agnostic reliability layer that separates candidate ranking from exact verification and explicit abstention, and exact conjunction over represented event constraints and linked source-record connectivity can improve selective reliability, while parser quality and candidate retrieval re...
Yu-Teng Sun, Yang Su, Xu-An Wang· Computers, Materials & C...· 0 citations
It is suggested that claim-level verification produces a useful reliability signal, and that calibration is necessary to convert that signal into trustworthy confidence estimates for selective answering, and that calibration is necessary to convert that signal into trustworthy confidence estimates for selective answeri...