Enhancing AI Agent Evaluation through Critical Step Identification

Abstract

The growth of AI based mobile agents due to recent improvement in large language models (LLMs), has unraveled new possibilities for automating user tasks (sending emails, booking cabs) on mobile devices. The accurate evaluation of AI agents presents a significant challenge, while recent benchmarks and datasets aim to assess agent performance, mostly rely on static or last k step-based evaluations and often overlook the nuanced, step wise decision processes essential for task completion in real world scenarios. To overcome this limitation, we introduce a novel evaluation system that leverages “Sentence Bidirectional Encoder Representations from Transformers (S-BERT)” cosine similarity to identify and prioritize key steps in multi modal agent execution traces. Our framework systematically compares semantic alignment between user intent and agent observations at each step across diverse tasks, applying dynamic thresholding methods to select the most relevant execution segments. By computing the cosine similarity between the agent's observations and the user's task, and then selecting the top 30% of steps based on similarity scores, we achieved comparable evaluation accuracy (90% closer to human evaluation)-this process enables efficient, high-fidelity evaluation using only critical steps and associated screenshots, reducing the computational burden and token usage when interfacing with LLMs for outcome verification. Our methodology provides a practical solution for comprehensive mobile agent benchmarking and establishes new directions for efficient, context aware autonomous agent evaluation.