Skip to content
Preprint

VTR-Bench: A Systematic Benchmark for Evaluating Visual Text Rendering in Video Generation

Oct 2026 · 3 citations · 56 references
Computer Science

TL;DR

An automated evaluation pipeline with human alignments that separately assesses text fidelity through carrier-specific transcription and scene and motion requirements through a prompt-specific chain of query and a Director agent coordinates image and video generation with visual evaluation, guiding iterative refinement and candidate selection through visual feedback.

Abstract

Recent video generation models can produce highly realistic videos from natural language instructions, with visual quality approaching cinematic standards. Existing evaluation benchmarks, however, predominantly assess visual quality, aesthetic appeal and physical plausibility, while paying limited attention to text, an essential medium for conveying information in everyday scenes. A generated video may appear visually compelling and feature lifelike subjects, yet still render the text within the scene incorrectly. To address this overlooked dimension, we introduce \textbf{VTR-Bench}, a systematic benchmark for evaluating the \textbf{V}isual \textbf{T}ext \textbf{R}endering capabilities of video generation models. VTR-Bench situates text within concrete application scenarios, such as advertisements and scientific videos, with 300 carefully constructed prompts spanning five scenario categories. We develop an automated evaluation pipeline with human alignments that separately assesses text fidelity through carrier-specific transcription and scene and motion requirements through a prompt-specific chain of query. Beyond evaluation, we introduce a \textbf{Keyframe-Guided Agentic Framework} in which a Director agent coordinates image and video generation with visual evaluation, guiding iterative refinement and candidate selection through visual feedback. Experiments on 11 state-of-the-art models reveal widespread difficulties in accurately rendering scene text, with the best-performing model recording an overall word error rate (WER) of 0.250. We further analyze text rendering failures to characterize the challenges faced by current video generation models. These findings highlight visual text rendering as a key challenge for video generation and demonstrate a practical path toward improvement. Code is available at https://github.com/hardenyu21/VTR-Bench.

View source

Similar papers

Sep 2026

Beyond Legibility: Benchmarking Visual Text Rendering and In-Place Editing in Unified Video Generation

A video can exhibit convincing motion and photorealism yet fail immediately when visual text collapses. Unlike generic scene content, visual text is unforgiving in video generation: minor stroke corruption, temporal instability, or editing errors instantly break legibility and realism. Existing benchmarks overlook this...

Zi-Ying Zhang, Li-Tao Li, Junchao Liao et al. · 1 citation · ⚡1
Preprint Oct 2026

TextVBench: Evaluating Text Rendering and Preservation in Video Generation

TextVBench is introduced, a benchmark and evaluation framework that formulates visual-text evaluation as a multi-target, multi-instance spatiotemporal correspondence problem, and is organized around controlled text, layout, carrier, and motion attributes so that aggregate scores can be traced back to specific capabilit...

Rui-Qiang Zhang, Heng-Yi Wang, Chang Liu et al. · 0 citations
#artificial intelligence Review Oct 2026

UltraText Bench: A Comprehensive Bilingual Benchmark for Evaluating Visual Text Rendering in Image Generation

Dense visual text requires image generators to reproduce long strings across multiple regions with correct placement and legibility. As short-string rendering improves, evaluation must test sustained performance across more demanding scenes. We introduce UltraText Bench, a bilingual benchmark for prompt-only generation...

De-Yuan Liu, Yihao Hu, Jing-Xuan Zhang et al. · 0 citations
#artificial intelligence Review Sep 2026

ViTeX-Bench: Benchmarking High-Fidelity Video Scene Text Editing

Recent video generation is increasingly realistic and controllable, yet video editing remains less developed, particularly for precise local edits that must preserve the original scene dynamics. Video scene text editing replaces text on scene surfaces, such as storefront signs, whiteboards, and product labels, while pr...

Xing-Hao Chen, Xiang-Bo Gao, Jiong-Ze Yu et al. · 1 citation
Preprint Oct 2026

Towards Unified Evaluation of Prompt Enhancers for Video Generation

This work introduces PEBench, the first unified benchmark for direct PE evaluation across text-to-video, image-to-video, and reference-to-video prompt enhancement, and develops PEBench evaluation, an evidence-grounded framework that combines modality-aware fact extraction with rubric-based assessment across 24 criteria...

Ya-Wen Shao, Yu-Bo Zhu, Zi-Yun Dai et al. · 1 citation
Sep 2026

WanPE: Towards Cinematic Prompt Enhancement for Modern Text-to-Video Generation

WanPE formulates shot-level cinematic plans via video-grounded reverse construction and employs Semantic-Consistency GRPO (SC-GRPO) to faithfully preserve user requirements across shots and over time, leading all evaluated commercial offerings at 5-15 seconds and remains competitive with Seedance 2.5 at 30 seconds.

Yu-Bo Zhu, Ya-Wen Shao, Zi-Yun Dai et al. · 4 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.