A case study of an NLP system for email-based customer support evaluated both intrinsically and extrinsically via a before-and-after study in deployment highlights a mismatch be-tween benchmark performance and real-world effectiveness.
The paper argues for evaluation procedures that disclose the tested configuration, validate questions and successful task completion, report performance alongside cost and execution time, and make the scope of generalization explicit, and discusses Isotanta, a crowdsourced benchmarking platform, as a practical example...
Organizations increasingly use frontier language models to analyze customer feedback, but answer quality also depends on how that feedback is organized and made available. We define a \emph{customer context graph} as a unified model of customer and business context. Typed relationships connect customer objects (feedbac...
G. Raviraja, Viraj Bagal, Prabhath Chellingi· 0 citations
This work introduces an automated pipeline for generating synthetic datasets of emails reflecting realistic workplace scenarios, along with long- and short-form questions and gold answers grounded in the data, and underscores the importance of realistic, high-complexity evaluation data for developing stronger real-worl...
Amey Varhade, Ananya Sutradhar, Ravishankar Krishnaswamy et al.· 0 citations
Large language models (LLMs) are increasingly used through terminal user interfaces (TUIs), yet task completion alone does not capture how difficult an interface is to understand and operate. Existing human assessments and LLM-generated ratings or reports do not provide repeatable measurements of interaction effort gro...
Rui-Da Hu, Yuan-Hao Wang, Chao Peng et al.· 0 citations
AI agents are increasingly being programmed to automate knowledge work over large collections of unstructured data. Such automation requires repeatability: when the underlying evidence is unchanged, the agent's categories, priorities, and counts should not shift materially between runs, even if each individual answer a...
This work studies whether task-specialist models should be handled by separate task-specialist models or by a single model trained through multi-task training, sequential updates, or model merging, and concludes with practical guidelines for selecting fine-tuning strategies in real-world settings.
