Skip to content
Open access

Structurally conditioned counterfactual generation and validation for question answering from scientific text

Oct 2026 · Frontiers of Computer Science · Vol 8 · 0 citations · 24 references

TL;DR

The study evaluates generation quality, intervention sensitivity, and cross-stage consistency in a large language model-based pipeline for structured processing of scientific abstracts and for generating paired factual and counterfactual question-answer (QA) instances under document-level intervention constraints.

Abstract

A large language model (LLM)-based pipeline is presented for structured processing of scientific abstracts and for generating paired factual and counterfactual question-answer (QA) instances under document-level intervention constraints. The implemented system extracts entities, events, and relations; induces a lightweight structural causal model (SCM) over scientific variables such as method, data, intervention, outcome, and metric; generates five factual QA pairs per abstract; rewrites them under one selected intervention; and filters inconsistent counterfactuals. The source export contains 100 Scopus abstracts, of which 67 documents with complete pipeline artifacts form the validation cohort. Generation uses gpt-4.1 , while automated validation uses gpt-5-mini under a 15-criterion checklist; three human annotators independently score the same 67 documents. Human agreement is moderate, with a mean pairwise Cohen's kappa of approximately 0.54. Automated validation is high for original QA and relation-level correctness (0.97 and 0.89) but very low for document-level global consistency (0.07), revealing a substantial gap between local correctness and cross-stage coherence. Under the score-based intervention analysis, 25.4% of documents exhibit a validation decrease above the predefined threshold. We refer to this quantity as the intervention degradation rate rather than as bias propagation, because the interventions concern scientific-content variables rather than protected attributes. The study evaluates generation quality, intervention sensitivity, and cross-stage consistency; it does not train a foundation model or claim training-time bias mitigation.

Read PDF

Similar papers

#small language model Review Open access Oct 2026

Diagnosing the limits of large language models for structured scientific evidence extraction

A schema-grounded, tiered evaluation framework for structured scientific evidence extraction that increases task difficulty from single-entity identification to relational tuples and corpus-level statistical tasks and provides a reusable query suite and diagnostic evaluation framework to support future fine-grained ben...

Zhiyin Tan, Jennifer D'Souza, Sören Auer · 0 citations
Open access Sep 2026

Improving Reliability and Explainability of Medical Question Answering Through Atomic Fact-Checking in Retrieval-Augmented Large Language Models: Creation and Validation Study

Abstract Background Large language models (LLMs) exhibit extensive medical knowledge but are prone to hallucinations and show low fact-level explainability, limiting clinical adoption and regulatory compliance. Existing approaches, such as retrieval-augmented generation, partially address these issues by grounding answ...

J. Vladika, A. Domres, Mai Q. Nguyen et al. · 0 citations
#artificial intelligence Review Open access Oct 2026

SurveyGraph: Generating Academic Surveys via Methods-Entity Knowledge Graphs

Most automated survey generation systems retrieve papers as flat documents, limiting their ability to capture research structure. We present SurveyGraph, a framework built around a Methods-Entity Knowledge Graph (MEKG) encoding method-level follow-on, competition, and shared-resource relations. The graph supports core-...

Yi-Fan Yuan, Yan-Cheng Zhu, Zuo-Yu Zhang et al. · 0 citations
Open access 2026

EviGraphRAG: Constraint-Verified Retrieval-Augmented Question Answering over Event Knowledge Graphs

EviGraphRAG is proposed, a ranker-agnostic reliability layer that separates candidate ranking from exact verification and explicit abstention, and exact conjunction over represented event constraints and linked source-record connectivity can improve selective reliability, while parser quality and candidate retrieval re...

Yu-Teng Sun, Yang Su, Xu-An Wang · 0 citations
Open access Sep 2026

Improving reliability of large language models via claim-level self-verification and uncertainty calibration

It is suggested that claim-level verification produces a useful reliability signal, and that calibration is necessary to convert that signal into trustworthy confidence estimates for selective answering, and that calibration is necessary to convert that signal into trustworthy confidence estimates for selective answeri...

Abhigyan Pal · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.