Introducing Safe LLM-Agent Execution via Typed Tool Mediation and World-Effect Evaluation

Abstract

LLM-based agents are increasingly used as action-taking systems that operate over files, databases, networks, and external services. In this setting, safety failures can become unauthorized world effects, such as data deletion, protected-resource modification, or secret exfiltration, rather than merely unsafe text outputs. We present AgentOS, a Buildroot-based execution environment for safe tool-using LLM agents. Its core component, the Verified Toolchain Reference Monitor (VTRM), mediates model-generated tool requests before execution by converting untrusted model outputs into typed requests and enforcing capability-aware and label-aware decisions over filesystem, database, and network operations. AgentOS further includes a host-side World Effect Judge (WEJ) for evaluation. WEJ measures task completion and safety using filesystem snapshots, database diffs, audit logs, and network observations rather than model self-report. We evaluate AgentOS on benign and adversarial task suites using two LLMs. Across both models, AgentOS matches prompt-only and naive baselines on benign-task utility. On adversarial tasks, AgentOS blocks all evaluated harmful cases, while both baselines leave substantially more unsafe effects unblocked. We also model the core VTRM decision path in TLA+ and check representative confidentiality and fail-closed properties. These results suggest that typed tool mediation and host-side world-effect evaluation can improve LLM-agent execution safety while preserving benign-task utility. The source code and documentation are publicly available under the MIT License at https://github.com/zhoujune/AgentOS, with an archived artifact at https://doi.org/10.5281/zenodo.20046861; the demonstration screencast is available at https://youtu.be/bqJsTRJbIkU.