Free-text clinical records represent an untapped wealth of data for secondary use, but realising their potential is limited by resource demands necessary for accurate information extraction at scale. We introduce a scalable, resource-efficient, and high-performance information extraction pipeline that leverages large language models (LLMs) to address these challenges. Our pipeline was developed and tested using real-world dual specialist-annotated ophthalmic clinical letters, and achieved strong performance with a proprietary model in development, yielding a maximum micro-averaged F1 score of 0.954 (95% CI 0.941–0.967) for diagnosis across nine conditions through iterative prompt refinement alone, also demonstrating strong generalisability (micro-F1 0.945–0.980) in temporal validation. This approach was extended to other models in the same family and 17 LLMs from seven open-weight LLM families. Beyond performance, we develop a multi-dimensional assessment for deployment in data extraction tasks, including an error taxonomy and Pareto frontier analyses to systematically map the operational trade-offs across different LLM configurations. A robust approach to operationalisation in real-world workflows at scale may help lay the foundation for next-generation data pipelines that accelerate scientific discovery and power continuous learning health systems.
A. Y. Ong, Quang Nguyen, I. Barai et al.· npj Digital Medicine· 1 citation
Agentic applications are shifting AI serving from isolated model inference to long-running workloads in which LLMs coordinate tools, environments, and persistent state. However, the system behavior of these workloads---where latency, cost, and bottlenecks arise---remains poorly characterized, leaving serving systems to rely on assumptions built for conventional inference. We present AgentSysBench, a benchmark suite and measurement toolkit with ten representative agentic applications and unified systems-level instrumentation. Across controlled deployments and production traces, we identify six properties that distinguish agentic workloads from conventional LLM serving: (1) execution is heavyweight and stateful, with non-LLM components dominating latency in 5 of 10 applications and sandbox working-set memory peaking at 28 GB per session; (2) applications compose components with heterogeneous resource affinity---GPU-bound inference, memory-bound retrieval, CPU-bound sandboxes---whose task latencies diverge by up to 32x; (3) bottlenecks shift across requests, models, and deployments; (4) production sessions hold state idle for minutes to hours between active steps; (5) a control-plane tax---auxiliary LLM calls and context overhead from tool schemas and observations---crowds out productive compute and context; and (6) production traces from three applications reveal heavy cross-request redundancy in search queries and web fetches, exposing a large caching opportunity. Four design explorations demonstrate that these findings are actionable: task-aware serving reduces latency by 29--40%, communication-aware placement by up to 4.5x, state offloading reduces memory usage by 4.6x, and tool-result caching removes 35.2% of redundant search calls and saves 19.3% of aggregate search latency.
Chaokun Chang, Yukun Zhou, Kaihua Fu et al.· 1 citation
While AI agents delivered highly efficient, directionally aligned assessments, they did not fully capture the nuances of human clinical judgment and could not substitute for physician-centered evaluation and promise assistive tools that can triage or pre-screen outputs to reduce human burden.
Peilun Shi, Jian Li, Ziqi Yang et al.· npj Digital Medicine· 0 citations