Skip to content
Open access

Context Matters in LLM-Assisted Qualitative Data Analysis: Workflow Development and Multidimensional Evaluation in Health Research

Sep 2026 · medRxiv · 0 citations
Medicine

TL;DR

A context-specific workflow was developed that translated framework analysis into bounded, sequential LLM-supported tasks and applied it to 20 Chinese-language interviews on fear of dementia, where Qwen-output showed the most consistently favourable overall profile, while Claude-output was most often ranked first in individual domain-dimension evaluations.

Abstract

Large language models (LLMs) are increasingly used for qualitative analysis, but strong language performance does not guarantee strong interpretation when meaning depends on method and context. We developed a context-specific workflow that translated framework analysis into bounded, sequential LLM-supported tasks and applied it to 20 Chinese-language interviews on fear of dementia (approximately 150,000 characters). Four multilingual LLM-generated outputs and a previous researcher-only analysis were blindly evaluated by 16 relevant experts across different analytical domains and qualitative quality dimensions (496 expert-item records). All LLMs completed the workflow, but rankings varied markedly. Qwen-output showed the most consistently favourable overall profile, while Claude-output was most often ranked first in individual domain-dimension evaluations. The researcher analysis received both highly favourable and highly unfavourable rankings; no output was consistently preferred. LLM-assisted qualitative analysis should be evaluated within its research contexts, with transparent workflows, multidimensional assessment, and continued researcher oversight.

Read PDF

Similar papers

#artificial intelligence Review Sep 2026

A primer on evaluation methods for large language models in healthcare

This article aims to provide a pragmatic basis for designing and executing rigorous evaluations of healthcare LLMs by describing underlying concepts and potential pitfalls, while emphasizing the importance of aligning evaluation methods with the research question.

S. E. McKinney, Phuc Vu, Samuel A. Justice et al. · 0 citations
Review Open access Sep 2026

Evaluating large language models as clinical decision support tools in primary healthcare settings: Protocol for a multi-country comparative validation study on expert-adjudicated hypothetical vignettes (hypMOOVE-PHC)

The hypMOOVE-PHC study is the hypothetical vignette phase of the Massive Open Online Validation and Evaluation (MOOVE) initiative, implemented in Kenya, Malawi, and Tanzania, and aims to validate a pool of LLMs through clinical review of expert-generated vignettes through fully crossed repeated-measures comparative eva...

P. Macharia, C. Kachimanga, M. Mahende et al. · 0 citations
Review Open access Oct 2026

Sociotechnical analysis of qualitative data using a large language model-assisted framework with performance compared to a human-only coded reference standard.

Objective This methodological validation study evaluated a large language model (LLM), multi-step framework for thematic analysis (TA) of healthcare interviews, compared to a human-only coded reference standard. Materials and Methods The unit of analysis was deidentified transcripts of interviews with geriatrics pati...

J. Butler, Makoto Jones, C. Balbin et al. · 0 citations
#explainable ai Review Open access Sep 2026

Large language models in healthcare: applications, evaluation frameworks, and governance pathways — a scoping review and multidimensional framework

Current evidence supports cautious deployment of LLMs in selected healthcare tasks under structured oversight, and the proposed five-dimensional framework coupled with a three-tier risk model is intended to support researchers and healthcare organisations in assessing readiness and implementing LLM-enabled tools respon...

J. C. Ferreira, Isabel Rosa · 1 citation
Review Aug 2026

Evaluating hybrid human-LLM coding workflows for qualitative research in medical education: A generalizability study.

D-study simulations support the use of hybrid human-LLM workflows to reach coding consensus for deductive reasoning tasks through mode responses, an alternative consensus-generating mechanism to traditional adjudication discussions.

Emily Rush, M. Karim, George S. Yacu et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.