FaithEyes: Towards Faithful Tool Use via Multi-Agent Process-Image Verification

Haoqing Wang Xing-Run Xing Wei Xia Zi-Heng Li Ye-Hui Tang

Jul 2026 · arXiv.org · Vol abs/2607.28225 · 1 citation · 49 references

Computer Science

TL;DR

FaithEyes, a multi-agent self-judging framework that uses a VLM to judge whether each process image helps answer the question and designs a multi-agent framework where the model itself serves as a subagent to judge the tool calls from the main agent, eliminating any dependence on external models at inference.

Abstract

Agentic vision-language models (VLMs), which interleave textual reasoning with explicit tool calls such as cropping and code-based image manipulation, have emerged as a compelling paradigm for reliable and interpretable multi-modal reasoning. However, recent studies have revealed that such models often use tools unfaithfully. Many process images are irrelevant to the question (e.g., the crops miss the queried target), yet the tool call still receives full credit and the model still answers correctly. Such decorative or misaligned tool calls waste computation and reveal that the model does not faithfully use the evidence it retrieves. This may stem from two limitations of prevailing methods: the tool reward fails to distinguish useful from useless calls, and tool feedback carries no signal of usefulness. To this end, we introduce FaithEyes, a multi-agent self-judging framework. Concretely, we use a VLM to judge whether each process image helps answer the question. The judgement is injected into the reasoning context as part of the tool observation to help subsequent reasoning, and meanwhile is used to scale the tool reward by the helpful-tool ratio to suppress reward hacking. To keep judgement available at evaluation, we further design a multi-agent framework where the model itself serves as a subagent to judge the tool calls from the main agent, eliminating any dependence on external models at inference. Training via a two-stage SFT + RL pipeline on adapted open-source data, FaithEyes attains competitive or superior accuracy across visual perception and reasoning benchmarks, while substantially improving tool faithfulness and reducing inference cost. The homepage is at https://github.com/Mosi-AI/FaithEyes.

DOI arXiv Semantic Scholar

View source

Similar papers

#artificial intelligence Preprint Sep 2026

Making Every Tool Call Count: Necessary Tool-Evidence Path Rewards for Agentic Vision-Language Models

This work proposes NTEP-R (NTEP Reward), a supervision mechanism ensuring that each tool invocation strictly advances the reasoning process toward the final solution, and introduces a non-repeated-goal regularizer to penalize redundant calls that revisit satisfied NTEP goals.

Xing-Ming Long, Yu Liu, Zhi-Wei Yang et al. · 0 citations

Preprint Aug 2026

Self-Evolving Code-with-Image Reasoning

Multimodal models increasingly reach for tools when solving visual tasks (crop, zoom, rotate, brighten), a paradigm known as thinking-with-images. The central challenge is one of perception: tools mostly serve to expose visual evidence, reasoning over that evidence stays in language, and most targets are ones a human c...

Tianze Yang, Liang Wu, Rui Sun et al. · 1 citation

Preprint Aug 2026

VISA: Agentic Self-Evolving Data Synthesis for Multimodal Instruction Following

VISA (Visual Instruction Synthesis Agent), an agentic framework that reformulates multimodal instruction synthesis as a self-evolving loop, consistently improves multimodal instruction following over strong baselines, while preserving general multimodal capability across seven public benchmarks.

Min Zeng, Guanxin Tan, L. Cen et al. · 1 citation

Preprint Aug 2026

Forgotten in Weights, Recovered by Tools: Agentic Tool Unlearning for LLM Agents

Experiments on RWKU and MUSE across different LLM architectures show that ATU achieves a better balance between target forgetting and retained utility, making unlearning more robust under tool-augmented agent deployment.

Baicheng Chen, Zhe-Yuan Liu, Jingyu Zhang et al. · 0 citations

#artificial intelligence Preprint Sep 2026

Eliciting Self-Verification in Multimodal Reasoning Agents with Reinforcement Learning

Self-Verification via Reinforcement Learning (SVRL), an RL-only finetuning framework that trains multimodal agents to verify and filter retrieved evidence within their own reasoning traces, reducing reliance on external verifiers at inference time is presented.

Vishwas Sathish, Viresh Ranjan, Xin-Liang Zhu et al. · 1 citation

Preprint Aug 2026

OpenVisTool: An Open Recipe for Synthesizing Instructive Visual Tool-Use Trajectories

Visual tool use has emerged as a fundamental capability for multimodal agents to actively acquire evidence beyond a fixed image encoding. The prevailing recipe learns this capability from teacher-generated trajectories filtered for answer correctness, implicitly assuming that every successful demonstration provides eff...

Changhao Xiang, Shi-Lin Zhang, Zheng Ma et al. · 1 citation