Background: Medical artificial intelligence (AI), machine learning (ML), and deep learning (DL) studies frequently begin with datasets collected for routine care rather than for computational modeling. Such datasets may contain inconsistent variables, heterogeneous measurement time points, unexplained NaN values, poorly defined outcomes, missing metadata, and insufficient documentation, which can compromise model development before any algorithm is selected. Methods: This Technical Note proposes a physician-facing Clinical AI-Readiness Guide for preparing medical datasets before AI-based analysis. The guide was developed as a practical framework organized around pre-modeling decisions, including the clinical task, cohort, minimum common dataset, outcome definition, predictor variables, measurement timing, missing-data logic, standardization, non-tabular data linkage, data dictionary, and validation readiness. Results: The proposed guide translates AI-readiness principles into concrete data-collection rules for clinical, laboratory, imaging, physiological-signal, textual, follow-up, and multimodal data. It emphasizes clinically consistent data acquisition, reliable target labeling, explicit missing-data logic, patient-level linkage, structured metadata, and validation feasibility. A structured checklist and scoring approach are also proposed as practical pre-modeling assessment tools to classify datasets as not ready, exploratory only, ML-ready with limitations, or AI-ready for model development. Conclusions: Medical AI-readiness should be established before model development begins. By helping physicians collect, structure, and document data more consistently, the proposed guide may improve collaboration between clinical and technical teams and reduce preventable dataset-related failures in medical AI research.
Cătălin Anghel, A. Anghel, M. Craciun et al.· Journal of Clinical Medicine· 0 citations
Background: Evaluation of LLM-based automated grading often relies on comparison with a single human score, which can obscure meaningful variability among raters of open-ended answers. This study introduces ConsensusGrade, a consensus-aware framework that treats the human reference as a scoring envelope rather than as a single point. Methods: We analyzed 1000 open-ended student answers from 100 students across 10 questions, each graded by four evaluators. Six previously generated and aligned automated grading configurations from GradeAgentOps were compared with the four-rater human reference. The score sets were generated using Llama 3.3 70B Instruct as the primary grader, with Qwen 2.5 14B Instruct for semantic repair. Results: Human evaluators showed meaningful agreement, with ICC(A,1) = 0.712, but exact four-rater agreement occurred in only 2.2% of records. Broad score dispersion occurred in 59.0%. All automated configurations showed negative bias relative to the human median. FULL achieved 68.5% inside-envelope positioning and a chance-adjusted score of 0.454; under the central-trimmed envelope, this rate decreased to 34.3%, while configuration ordering was preserved. Conclusions: ConsensusGrade provides a diagnostic framework for interpreting automated scores relative to observed human variability; inside-envelope rates should not be interpreted as stand-alone measures of grading accuracy.
Cătălin Anghel, A. Anghel, Mihai Vlase et al.· Applied System Innovation· 0 citations
Background: Large language models (LLMs) are increasingly explored for automated educational assessment, while future educational platforms may combine grading, feedback, learner analytics, and personalization. The objective of this study was to determine whether student-history metadata can influence the numerical score assigned to the same answer. Methods: This study introduces GradeDrift-LLM, a controlled framework for measuring student-history-induced score drift in LLM-based automated grading. We evaluated 1000 Computer Science answers from 100 students across six student-history conditions and eight open-weight LLMs. For each grading instance, the submitted answer, question, reference answer, rubric-related information, scoring scale, and grading instruction were kept constant; only the student-history condition varied. Results: Across 39,997 valid paired comparisons, 83.92% showed no drift, 9.40% showed upward drift, and 6.68% showed downward drift. Mean absolute drift was 0.2137 points, and the 95th percentile absolute drift was 1 point. Positive-history frames tended to increase scores, whereas negative-history frames tended to decrease them. Drift was model-dependent, not uniformly explained by approximate scale, and present in both technical and argumentative answers; rare extreme deviations reached 10 points. Conclusions: Student-history metadata can influence LLM-generated grading scores despite explicit instructions to ignore it. Future LLM-based grading systems should separate answer-based scoring from learner-context-based personalization and validate score invariance under controlled learner-context variations.
Cătălin Anghel, A. Anghel, M. Craciun et al.· Machine Learning and Knowled...· 0 citations