Aug 2026· International Journal of Creative and Open Research in Engineering and Management· 0 citations
TL;DR
The results indicate that agentic orchestration combined with permission-based safety controls provides a practical foundation for autonomous, trustworthy task automation across operating systems and web environments.
Abstract
Conversational AI assistants such as Siri, Google Assistant, and ChatGPT have improved how users interact with digital systems, yet they remain confined to answering questions rather than completing real-world tasks. Users must still manually open applications, navigate websites, fill out forms, and switch between software to finish even simple workflows, which limits productivity and creates accessibility barriers for elderly users, first-time computer users, and individuals with disabilities. This paper presents TEXA OS (Trusted Executive Assistant), a self-improving agentic artificial intelligence platform designed to function as a complete digital executive assistant rather than a conversational chatbot. Unlike static automation scripts built with tools such as Selenium or Playwright alone, TEXA OS combines natural-language intent understanding, autonomous task planning, and multi-agent execution to decompose a single spoken or typed request into an ordered set of executable subtasks.
A FastAPI-based AI orchestrator interprets user intent using a large language model and delegates the resulting subtasks to specialized execution agents responsible for browser automation, operating system control, document generation, and communication automation. The system is built on a React and TypeScript frontend with continuous voice-recognition support through the Web Speech API and a PostgreSQL backend that
persists user preferences, task history, and long-term AI memory, allowing the platform to adapt future task execution based on prior interactions. A distinguishing capability of TEXA OS is its Website AI Navigation Module, which allows users to retrieve information from complex websites—government portals, banking systems, and educational.
The system was evaluated across unit, integration, and system-level test cases spanning voice recognition, browser automation, document generation, and permission-gated execution, achieving an overall task-execution success rate of approximately 96% and an overall system reliability of approximately 86%. The results indicate that agentic orchestration combined with permission-based safety controls provides a practical foundation for autonomous, trustworthy task automation across operating systems and web environments. Keywords: Agentic AI, Task Automation, Large Language Models, Browser Automation, Voice Recognition, AI Orchestrator, Natural Language Processing, FastAPI, Self-Improving Systems, Human-Computer Interaction.
JarvisBench, a benchmark for measuring the dual value of mediation in long-horizon agent workflows, is introduced and preliminary results suggest that Jarvis-style mediation can provide trace-grounded responses to user questions and improve task performance when sparse user guidance is injected at appropriate moments.
The resulting prototype confirms that a cloud-hosted multimodal LLM, when combined with a minimal and well-structured web stack, can serve as a practical foundation for next-generation digital assistants suitable for customer support, education, and personal productivity applications.
G. Vamsi, Vinay Kumar Male· International Scientific Jou...· 0 citations
The point of the project is less to compete with Siri or Alexa than to show that a legible, few-thousand-line assistant is enough to open apps, type into them, run searches, coordinate concurrent actions, recover from single-step failures, and be driven entirely from a phone in another room -- without the LLM ever touching the keyboard.
Recent advances in large language models have enabled a new class of agentic data science systems that allow users to complete complex data science workflows through natural language. Although these systems can significantly reduce manual effort, it remains difficult to diagnose their behavior and steer the reasoning process when failures or unexpected outputs occur. We present MUSE, an interactive meta-agent that enhances user understanding and control of agentic data science systems by (1) dynamically restructuring low-level execution traces into multiple semantic levels that support navigation from high-level overviews to low-level implementation details; (2) enabling users to reference specific workflow steps in context to ask grounded questions, provide feedback, and revise problematic steps without manually locating relevant execution history; and (3) supporting mixed-initiative steering by surfacing suspicious steps for inspection, scaffolding the repair process, and translating user repair intent into contextualized instructions for the underlying agent. In a between-subjects study (n = 15), MUSE improved task efficiency and increased users'confidence in understanding and steering agentic data science workflows.
Wei-Hao Chen, Weixi Tong, Yuan Tian et al.· 0 citations
Digital Adoption Platforms (DAPs) are embedded overlays widely used on web systems to guide users through operations inside a page, helping them get started with unfamiliar interfaces quickly. Completing a real task, however, rarely means clicking a few buttons on a single page: it takes a sequence of actions that unfolds across changing page states. Prior studies have also treated automated web agent actions and guide text generation as two separate problems, and most of them feed models textual page representations such as the DOM or accessibility trees rather than the rendered screens that humans actually operate on. In this work we introduce MAG, the first benchmark that unifies task execution and guide writing into a single Multimodal Action and Guide task, with two grounding schemes over screenshots: Set-of-Mark element selection and raw pixel coordinates. We further build a complete harness for this compound task, covering annotation with LLM assistance and human verification, training, evaluation in live environments, and joint metrics for actions and guides. With this harness we evaluate frontier API models and open multimodal models, and report detailed analyses. Finally, we design a GRPO training method augmented with expert trajectories, which nearly doubles the success rate of a supervised 9B agent (from 6.9% to 13.2%) and improves guide quality at the same time. Even the strongest model completes fewer than 40% of the tasks, leaving ample room for future research.
Chengguang Gan, Hanjun Wei, Yunhao Liang et al.· 0 citations
Large Language Models (LLMs) are increasingly embedded into applications, allowing users to complete tasks either through direct manipulation or by delegating actions to conversational agents. However, little is known about how users balance these modalities when both are available. We present a web-based content management system augmented with an LLM agent through the Model Context Protocol (MCP), enabling users to perform CRUD tasks through a graphical interface, a conversational agent, or both. We conducted a between-subjects study (N=73) comparing three interaction modes: Traditional-Only, AI-First, and Hybrid. Across sixteen scenarios, we analyzed task completion time, interaction logs, and delegation behavior. AI-assisted interaction significantly reduced clicks, page navigations, and scrolling indicating lower interaction effort. Surprisingly, these reductions did not translate into faster task completion, as task duration did not differ significantly across conditions. We also found no significant relationship between CRUD operation type and delegation, suggesting that users did not systematically avoid delegating higher-risk actions. Instead, delegation varied far more between participants than between tasks, with individual differences accounting for roughly half the variance in assistant use (ICC = .50). Our findings suggest that the primary benefit of human--agent interfaces may be reducing interaction effort rather than improving speed, and that delegation reflects who the user is more than what the task demands.
Gavin Raine Dizon, Tyrone Justin Sta Maria, Jordan Aiko Deja et al.· 0 citations
