Evaluating enterprise AI assistants on everyday business tasks

Abstract

Enterprise AI procurement often combines a choice of application bundle with a choice of language model. This exploratory study examined three model selections available through enterprise interfaces: Opus 5.5 and GPT-6.1 Sol in the installed Microsoft 365 Copilot desktop application, and Gemini 4 Argon in the Gemini Enterprise web application. Six fictional English business scenarios addressed source authority, invoice arithmetic, customer communication, meeting actions, vendor eligibility and policy application. Each identical prompt was submitted three times in fresh chats for each selection, producing 54 retained responses and 18 responses per selection. Five binary checks were specified locally for each scenario before collection. All selections passed all 90 check opportunities and all 18 responses per selection passed five of five checks. However, supplementary inspection identified an unsupported procurement approval rule, an unstated assumption about missing revenue and a sentence-count deviation in Opus responses. These observations were outside, or not enforced by, the original operational scoring and were not used to revise the scores. Customer email lengths also differed descriptively while remaining within the requested range. The shared maximum score therefore indicates a ceiling of this small rubric; it does not establish model equivalence or error-free performance. The study used one unblinded AI assistant as operator and evaluator, lacked independent human ratings and did not control backend configuration. Its contribution is a transparent pilot dataset and an account of how narrow business checklists can conceal relevant differences. Enterprise purchasing decisions require further evaluation of actual workflows, review effort and application governance.