Chaitai Deb Purkayastha

Beyond Accuracy: Counterfactual Fragility and Demographic Bias in Clinical Evaluation of LLMs

Clinical LLM evaluation often emphasizes answer accuracy; however, accuracy alone does not test counterfactual consistency or demographic robustness. We evaluated six LLMs on 150 MedQA USMLE questions using two automated perturbation tests to assess their performance. The counterfactual validity (CFV) test asked each m...