As Large Language Model (LLM) agents have demonstrated broad competence, but they still struggle in specialized, real-world workflows. Existing approaches such as RAG, fine-tuning and tool integration improve knowledge access, model adaptation, and external functionality, yet they do not fully address a central gap: the absence of reusable procedural knowledge for carrying out domain tasks reliably. This paper examines the emerging notion of agent skills as a possible abstraction for addressing that gap. Agent Skills are modular packages of domain-specific procedural knowledge that can be injected at inference time. Intuitively, a skill is like a cooking recipe for an agent: it does not provide new ingredients or tools, but specifies how available resources should be combined to achieve a desired outcome. A community-driven skills ecosystem is already emerging at remarkable speed, with early evidence of meaningful performance gains across multiple domains. However, their value and limits remain open questions. We examine how skills may help address bottlenecks of current agents and how they may expand agent capabilities through reusable domain procedures loaded at inference time. We then outline open questions in skill construction, composition, evaluation, portability, governance, and security, and conclude with a call for contribution. Our goal is not to present skills as a settled solution, but to clarify their promise, limits, and the questions that must be answered before they can become a principled foundation for future agent systems.
Hanwen Xing, Haomin Zhuang, Xuandong Zhao et al.· Proceedings of the 32nd ACM...· 6 citations· ⚡1
Human evaluation of AI systems and digital products is costly, slow, and difficult to scale. Offline evaluations are more scalable but often abstract away human diversity and interactive behavior. We therefore introduce MatrAIx, a population-scale simulated-user evaluation infrastructure for testing AI systems and digital products with heterogeneous users. MatrAIx has three core components: First, Persona 8B contains 8.3 billion persona records represented by 1,290 categorical dimensions. Records are either sampled from a dependency graph that preserves correlated attributes or derived from human-authored profiles. We release a quality-filtered coreset of approximately 1 million personas, comprising 599,847 human-grounded and 400,000 synthetic records. Second, the MatrAIx Playground provides four environments in which diverse users evaluate and interact with digital products: Survey, AI Chatbot, Web, and App. Third, MatrAIx provides 1,010 application tasks spanning more than 25 domains, including Commerce, Software, Finance, and Healthcare. We conducted 18,189 evaluation trials across eight representative tasks. Persona agents were powered by three LLMs: Claude Opus 4.8, GPT 5.5, and Claude Haiku 4.5. The resulting feedback captures how decisions and preferences vary across persona backgrounds, including hesitation after a price increase, willingness to continue after an AI assistant fails, and latency tolerance. We conducted two main validation studies: First, a 400-trial controlled study evaluated persona adherence across ten behavioral attributes and all four environments. The declared behavior was expressed or correctly suppressed in 366 trials (91.5%). Second, human and LLM judges evaluated the extraction quality of human-grounded personas. Overall, MatrAIx provides an end-to-end infrastructure for evaluating AI systems and digital products with diverse simulated human users.
Xiaomin Li, Yuexing Hao, Jian Hou et al.· 0 citations