This paper maps key evaluation challenges and proposes three frameworks specific to these tasks and offers design teams a practical roadmap to incorporate continuous, evidence-based testing, promoting greater rigor and accountability in AI-assisted design practice.
Abstract
Large language models (LLMs) are being rapidly integrated into design workflows for Design Systems, UX research, and UX writing, but systematic methods to evaluate their safety and utility are lacking. Based on a scoping review, this paper maps key evaluation challenges and proposes three frameworks specific to these tasks. Each framework connects design activities to measurable success criteria (e.g., ≤ 3% hallucination, < 2s latency) and suggests datasets for reproducible testing. A demonstration with GPT-4o and Gemini 2.5 illustrates how the metrics identify the models' strengths and weaknesses. The guidelines offer design teams a practical roadmap to incorporate continuous, evidence-based testing, promoting greater rigor and accountability in AI-assisted design practice.
The rapid advancement of Large Language Models (LLMs) has created unprecedented opportunities for transforming educational program design. However, the lack of standardized evaluation frameworks raises concerns regarding the content quality and pedagogical effectiveness. This study proposes EduDesignEval, a comprehensi...
Ayanbek Serikov, A. Biloshchytskyi, A. Mukhatayev et al.· International Journal of Inf...· 0 citations
Multimodal large language models are quickly expanding what AI can support in STEM instruction and assessment, but they also undermine answer-focused, unsupervised assessments. We describe Project Ethel, an institutional platform for course-specific, instructor-controlled AI assistance that emphasizes controlled contex...
Gerd Kortemeyer· Journal of Physics, Conferen...· 0 citations
Although LLM-based benchmark ratings can approximate expert judgments and reduce the need for labor-intensive human triple coding, limitations remain regarding cost, genre- and task specificity, and sensitivity to text presentation and student grade level—factors that constrain immediate classroom use, particularly for...
ELBench is introduced, the first benchmark to evaluate all four requirements (General Capability, Safety and Trustworthiness, Basic Education, and High-Level Cultivation) on the same models under a common protocol, combining curated public sources with newly synthesized safety and cultivation data.
Yi-Lin Jiang, Xiao-Rong Zhu, Fei Tan et al.· 0 citations
This work designed a tool called dBlocks with the following features: blocks to scope content, a context manager to edit context, and inline execution to verify code, and shows how human-centered design can guide the development of LLM-integrated tools.
This one-day workshop provides hands-on, research-oriented training in fine-tuning large language models for extracting reliable, domain-specific insights from text data, and combines conceptual clarity with guided Jupyter notebooks, teaching participants how to fine-tune open-source models, apply efficient techniques...
Seraphine F. Maerz· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
