Skip to content
Log in
Open access

VQA-Guided Diffusion: Enhancing Text-to-Image Generation With Semantic Feedback From Visual Question Answering

Debashish Bhowmik Ishika Maity Ashis Kumar Pati
2026 · IEEE Access · Vol 14, pp. 115191-115210 · 0 citations · 32 references

TL;DR

It is shown that VQA can serve as an effective semantic feedback to significantly enhance prompt-image alignment without retraining the diffusion model, providing a powerful, interpretable and self-correcting strategy for text-to-image production.

Abstract

Diffusion models for text-to-image (T2I) synthesis, e.g. Stable Diffusion, generate visually realistic images, but often fail to capture the fine-grained semantic nuances of complex prompts. Typical failure modes include missing objects, incorrect attribute assignment e.g. wrong colors, distorted spatial relationships and inaccurate object counts. This paper proposes a VQA-Guided Text-to-Image Feedback Framework to address these limitations, substituting the traditional one-shot generation with an iterative self-correcting loop. In our framework, an input prompt is first decomposed into its semantic components (objects, properties, relationships) using natural language processing. Then a set of Visual Question Answering (VQA) questions is automatically generated stepwise to probe object existence, colors, spatial relations and quantities. A VQA model (BLIP) answers the questions on the initially generated image from Stable Diffusion. Whenever it detects a semantic mismatch, it automatically refines the prompt (for example, by adding explicit corrective instructions such as “ make sure the car is red ”), and feeds the revised prompt back into the diffusion model. This cycle is repeated until adequate semantic alignment is obtained, with the framework incorporating various assessment criteria (CLIP Score, VQA Accuracy, LPIPS, picture quality score, and consistency loss) integrated into a unified multi-metric evaluation framework. For our experiments on MS-COCO captions, DrawBench, and a custom set of challenging prompts, the proposed method resulted in the following improvements observed on our test sets: the average CLIP Score improved from 0.311 (baseline Stable Diffusion) to 0.364, a relative increase of 17%; VQA accuracy was above 87% for all prompts evaluated; and in a human preference study of 39 pairwise comparisons, the VQA-guided method was preferred in 76.9% of the cases. Ablation experiments demonstrated the beneficial contribution of many components of the framework. We show that VQA can serve as an effective semantic feedback to significantly enhance prompt-image alignment without retraining the diffusion model, providing a powerful, interpretable and self-correcting strategy for text-to-image production.

DOI Semantic Scholar
Read PDF

Similar papers

Preprint Jul 2026

TMI: Text-to-Image Meets Image-to-Image for Complementary Data Synthesis to Boost Long-Tailed Instance Segmentation

Large-vocabulary instance segmentation is constrained by long-tailed category distributions and fine-grained inter-class ambiguity. While data synthesis offers a promising alternative, current paradigms have complementary limitations: text-to-image (T2I) methods inherit noisy pseudo-labels and struggle on rare classes, whereas copy-paste methods compromise contextual realism. To address these issues, we propose a hybrid pipeline coupling T2I generation with context-aware image-to-image (I2I) editing. The T2I branch provides broad category and scene diversity, while a teacher-student scheme ensures label reliability by selectively retaining only prompt-specified categories. To strengthen supervision for rare classes, we introduce VRAIN (Verified Rare-class Augmentation via INstructed editing), a novel I2I editor. VRAIN inserts high-confidence instances at semantically appropriate locations within in-the-wild scenes, yielding semantically coherent and visually natural edits that reduce domain gaps and enable targeted augmentation. On the LVIS benchmark, our method surpasses existing baselines, improving overall AP by up to +4.0 points and rare-class AP by up to +9.5 points, while scaling effectively with backbone capacity. Our project page is available at https://seokhunchoi.github.io/TMI

H. Song, Seokhun Choi, Hoseok Do · 0 citations
#computer vision Preprint Aug 2026

When Generated Images Look Right and Retrieve Wrong: Coverage-Guided Cross-Scale Re-Indexing for Knowledge-Faithful Generative Perception

Multimodal information systems increasingly route generated visual content back through the same vision-language index that informed its production, so the output must remain retrievable by the queries it was meant to serve. When the scene contains entities at vastly different scales, existing language-guided generators condition on a single, globally pooled text embedding and quietly drop scale-specific concepts, breaking concept-query retrieval even when pixel fidelity is high. We formalise this failure as semantic collapse and propose CERES, a closed-loop multimodal indexing framework that builds a three-level semantic pyramid, mines implicit concepts via a co-occurrence-aware router, performs scale-routed cross-attention into a lightweight U-Net generator, and verifies coverage by re-indexing the generated image with the same frozen VLM. A continuously differentiable soft-Jaccard coverage objective returns dense gradients to the 0.39M-parameter generator under explicit non-degeneracy conditions, and coverage is verified by an independent DINOv2 linear probe trained only on external scene and object labels. On four pansharpening benchmarks across seven settings, CERES delivers the new state of the art with the largest gains where scale variation is most extreme. It also improves concept-query retrieval Recall@5 by +14.0 points and image-text mean reciprocal rank by 0.19 over the strongest baseline, showing that the closed loop preserves queryable content rather than self-referential feature consistency.

Guangyuan Dong, Chuang Liu, Yangchen Zeng et al. · 0 citations
Preprint Jul 2026

Show, Don't Tell: Evaluating Spatial Cognition in Generative Pixels Rather Than LLM Text

ProVisE (Protocolized Visual Evaluation), a benchmark-agnostic framework that elicits protocol-constrained visual answers from image-generation models and parses them into structured predictions compatible with original metrics, is proposed and revealed, revealing complementary strengths of pixel-space expression and text-based reasoning.

Xu Wang, Kaixiang Yao, Miao Pan et al. · 1 citation