Operationalizing Pragmatic Difficulty: Speech Act Hierarchy and the Appropriateness Gap in an AI-Generated Business English Oral Test

Abstract

Large language models can now draft scenario-based test tasks on demand, which raises a validation question that has attracted little attention: does the difficulty of the generated tasks behave as the construct predicts? This study examined an AI-generated oral test of three business speech acts to establish whether the tasks reproduced the theoretically expected ordering of difficulty, where within the rating framework difficulty was located, and whether the ability measured was separable from general business oral proficiency. Thirty business English undergraduates responded to twelve ChatGPT-generated tasks; two trained raters independently scored five performance dimensions, and a simulated BEC oral test supplied an external criterion. Scoring reliability was high (total ICC = .953; Cronbach’s α = .86). A 3 × 5 repeated measures analysis of variance revealed a significant effect of speech act (p = .004, partial η² = .213): apologies were significantly more difficult than requests and refusals, which did not differ from each other. A significant effect of rating dimension (p < .001, partial η² = .237) located the greatest weakness in pragmatic appropriateness and the least in discourse coherence, and a non-significant interaction indicated that this profile held across every act. Test totals correlated strongly but not perfectly with the external criterion (r = .778). The results suggest that construct-informed prompting can produce AI-generated pragmatic tasks with theoretically differentiated difficulty profiles, while pragmatic appropriateness emerged as a relatively stable area of weakness across speech acts.