Skip to content
Review Open access

Practical Guide to Large Language Models for Information Extraction in Behavioral Health Notes: Tutorial

Sep 2026 · JMIR Mental Health · Vol 13 · 0 citations · 27 references
Medicine

TL;DR

A schema-driven, zero-shot LLM workflow can support reproducible extraction of clinically relevant information from narrative notes and provides a transferable methodology for institutional adaptation and validation prior to clinical use.

Abstract

Abstract Background Mental health clinical notes contain decision-critical information often absent from structured electronic health record fields. Large language models (LLMs) can extract clinically relevant signals from narrative text; however, variability in output format, limited reproducibility, and inconsistent evaluation remain barriers to clinical deployment. Despite rapid advances in LLM-based information extraction, clear and reproducible guidance for interdisciplinary clinical teams is limited. Objective This tutorial aims to present a structured workflow for zero-shot information extraction from mental health clinical notes using locally deployed open-source LLMs. It aims to reduce barriers for clinicians and researchers with limited familiarity with natural language processing (NLP) or LLM-based pipelines. Each stage includes key decision points and examples. The workflow is illustrated on two tasks using synthetic notes: (1) detection of self-injurious thoughts and behaviors (SITB) in pediatric emergency department (ED) notes and (2) antipsychotic medication nonadherence detection in outpatient notes, using schema-constrained outputs and standardized evaluation. Methods We describe a five-stage zero-shot LLM pipeline: (1) infrastructure setup with local deployment via Ollama to prevent protected health information (PHI) transmission; (2) task definition specifying the clinical construct, output format, and evaluation; (3) dataset preparation using synthetic notes; (4) iterative prompt development using a hold-out development set with binary and Likert scale outputs constrained via JSON schemas; and (5) output parsing, normalization, and validation. We generated 300 synthetic notes per task using separate LLMs for generation and evaluation; 200 notes were used for evaluation, and 100 notes (50 positive and 50 negative) were used as a prompt-development set and excluded from final metrics. Evaluation used Large Language Model Meta AI (Llama) 3.2 and Llama 3.3 with deterministic decoding (temperature=0). Performance was assessed using accuracy, precision, recall, and F1-score; Likert thresholds were optimized using the Youden index with bootstrapped CIs. Results We demonstrated the pipeline’s functionality using 2 example behavioral health detection tasks. Across both examples, the more capable model (Llama 3.3) performed better than the lighter model used earlier in development (Llama 3.2), and we described how the pipeline’s evaluation and error-analysis steps work in practice. These examples also illustrated 2 useful design choices: requiring the model to output in a fixed format reduced errors, and using a graded rating scale, rather than a simple yes/no format, allowed the detection threshold to be adjusted based on clinical risk tolerance. These results are meant to show that the pipeline works as intended, not to serve as a benchmark of real-world accuracy. Conclusions A schema-driven, zero-shot LLM workflow can support reproducible extraction of clinically relevant information from narrative notes. Local deployment enables processing without transmitting PHI to external servers. This tutorial provides a transferable methodology for institutional adaptation and validation prior to clinical use. All prompts, code, and datasets are publicly available via Zenodo (European Organization for Nuclear Research [CERN]).

Read PDF

Similar papers

Review Open access Oct 2026

Guideline-grounded large language models for extracting genome-informed clinical recommendations from electronic health records.

BACKGROUND Precision medicine requires delivery and tracking of genome-informed risk assessments (GIRAs), often documented in unstructured electronic health record (EHR) notes, making large-scale evaluation reliant on labor-intensive manual chart review. Large language models (LLMs) offer a promising approach to automa...

Yi Xin, K. W. Davis, Wu-Chen Su et al. · 0 citations
#large language models Review Open access Sep 2026

Large Language Models for Clinical Note Simplification: A Systematic Review and Experimental Evaluation of Medical Text Readability.

The findings suggest that conventional readability metrics should be extended with domain-specific measures to more accurately assess comprehensibility in medical texts and that large Language Models show strong potential to enhance the accessibility of clinical documentation for patients.

M. Teichmann, Pelin Özkara Menekseoglu, Julian Schwarz et al. · 0 citations
Open access Sep 2026

A Scalable Method for Validated Data Extraction from Electronic Health Records with Large Language Models

Health care organizations increasingly require structured, patient-level clinical variables for treatment decisions, operational workflows, quality measurement, and clinical trial screening. Relevant information is often fragmented across heterogeneous electronic health record (EHR) systems, unstructured formats, a...

Timothy J. Stuhlmiller, AJ Rabe, Jeff Rapp et al. · 0 citations
Review Open access Aug 2026

Large Language Models for Mental Health Prediction: Scoping Review of Bias and Clinical Utility Documentation in 2019-2024

Abstract Background A growing body of literature leverages large language models (LLMs) to make mental health predictions. However, these models are prone to bias, and studies to validate their clinical utility are lacking. Objective This scoping review aims to uncover bias and clinical utility limitations stemming fro...

Clémentine Bleuze, Karen Fort, Vincent P. Martin et al. · 0 citations
Review Sep 2026

Uncoded Clinical Features from Multilingual Electronic Health Records in Catalonia: Development and Validation Study

A privacy-preserving local hybrid framework combining a compact open-weight SLM with regular expression rules achieves high specificity and precision for extracting six uncoded clinical features from multilingual primary care narratives.

D. Ouchi, S. Riudavets, Á. Franquet et al. · 0 citations
Open access Sep 2026

Evaluating RAG Configurations for Clinical Information Extraction from EHR Notes: Aged Care Case Study

Empirical evidence is provided on how retrieval, embedding, and prompting choices influence clinical information extraction performance and practical guidance is offered for researchers and healthcare informatics practitioners seeking to optimise retrieval-augmented large language model systems for extracting clinicall...

Dinithi S. Vithanage, Quang Vinh Duong, Chao Deng et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.