Large language models (LLMs) are increasingly used to evaluate open-ended educational responses. However, their performance is often assessed using aggregate metrics that provide limited insight into prediction stability, uncertainty, error patterns, and feedback quality. This study presents EduFairBench, a reproducible evaluation protocol designed to characterize LLM behavior across short-answer assessment and automated essay scoring using open educational benchmarks. The protocol combines repeated inference, majority-vote consolidation, uncertainty estimation, error analysis, and structural evaluation of generated feedback within a unified experimental framework. Experiments were conducted on SciEntsBank, Beetle, and ASAP2, comprising 2,000 student responses and 10,000 independent LLM inferences. The results showed moderate predictive agreement with human assessment while revealing substantial differences between nominal and ordinal evaluation tasks. Repeated inference demonstrated high internal stability across benchmarks, although systematic errors remained in semantically adjacent categories, indicating that prediction consistency does not necessarily imply correctness. Feedback quality varied by task type, with longer textual contexts yielding more specific and pedagogically structured explanations. These findings demonstrate that evaluating educational LLMs requires complementary analyses beyond conventional performance metrics. EduFairBench provides a reproducible methodology for jointly analyzing predictive performance, robustness, uncertainty, and feedback quality, providing a comprehensive methodological framework for the rigorous evaluation of LLM-based educational assessment systems.
W. Villegas-Ch., Aracely Mera-Navarrete, Fernando Zúñiga-Tello et al.· Frontiers in Artificial Inte...· 0 citations
Phishing remains a prevalent cyberattack vector, and the widespread adoption of large language models (LLMs) has enabled adversaries to generate grammatically correct and contextually coherent phishing emails at scale, against which conventional detection systems are less effective. Although stylometric methods achieve over 95% accuracy within a single generator, their performance has not been systematically evaluated when the source model changes between training and deployment. This represents a significant gap, as adversaries can switch generators rapidly. A balanced corpus of 9,986 phishing emails was assembled, comprising 4,986 emails generated by three modern LLMs (GPT-4.1, DeepSeek 3.2, and Llama 3.3 70B) across five thematic categories, and 5,000 human phishing emails sampled in a stratified manner from five public sources. Seventeen stylometric features were extracted, and Logistic Regression and XGBoost classifiers were evaluated under intra-model, cross-model, threshold-recalibrated, cross-dataset, and aggregated-pool settings. Intra-model F1 scores reached 0.96 under stratified cross-validation and 0.999 on held-out splits used for the cross-model matrix. However, cross-model F1 dropped by 28.0 percentage points under the default decision threshold of 0.5. Notably, the area under the receiver operating characteristic curve remained above 0.96 in every off-diagonal cell, indicating that discriminative information is preserved even though the decision threshold is generator-specific. Recalibrating the threshold on a small target subset reduced the gap to 4.0 percentage points (an 86% reduction), and an aggregated-pool detector achieved F1 = 0.997 on each generator. This work reframes cross-model phishing detection from a problem of model incompatibility to one of practical calibration, and provides two deployable solutions, threshold recalibration on a small target slice and aggregated-pool training, along with a publicly released multi-LLM corpus.
Rommel Gutierrez, W. Villegas-Ch., Jaime Govea· Frontiers in Big Data· 0 citations
