Skip to content

Panoptic Scene Program Diffusion Transformer

Sep 2026 · 0 citations · 78 references
Computer Science

TL;DR

Under matched training and inference settings, PSP-DiT improves over a strong flat-text baseline across GenEval 2, SANEval-Simple, PSG-Score, and DetailMaster, with the largest gains on counting, attribute binding, role-sensitive relations, and long structured prompts.

Abstract

Modern text-to-image models produce high-fidelity images but still struggle with compositional prompts that require instance identity, attribute ownership, counting, spatial ordering, and role-sensitive relations. We introduce Panoptic Scene Program Diffusion Transformer (PSP-DiT), a diffusion-transformer architecture that treats a panoptic scene program as a first-class latent variable rather than an external control signal or post-hoc parse. PSP-DiT jointly denoises image latents and scene-program latents through coupled transformer streams, while panoptic grounding and cycle-consistency objectives tie object instances, attributes, relations, and counts to visual support in the generated image. Under matched training and inference settings, PSP-DiT improves over a strong flat-text baseline across GenEval 2, SANEval-Simple, PSG-Score, and DetailMaster, with the largest gains on counting, attribute binding, role-sensitive relations, and long structured prompts. The method preserves image quality, adds modest inference overhead, and remains robust to imperfect scene programs.

View source

Similar papers

#machine learning Preprint Oct 2026

ORCA: Hunting Compositional Failures in Text-to-Image Diffusion

Text-to-image diffusion models fail predictably on compositional prompts: attributes bind to the wrong objects, spatial relations invert, and multi-object scenes lose count. Recent architectures already augment CLIP with a T5 encoder precisely because CLIP's contrastive embedding loses compositional structure, yet thes...

Arshia Hemmat, Amirhossein Vahidi, Amitis Shidani et al. · 0 citations
Preprint Oct 2026

Not Just a Subject: Capturing Adaptive Scene-Specific Variance for Visual Storytelling

Recent advancements in diffusion models have significantly improved text-to-image generation, enabling their application to visual storytelling, where coherent image sequences are generated from narrative prompts. However, existing methods primarily focus on identity preservation and often fail to adapt to diverse scen...

SeungJu Cha, Ye-Chan Kim, Kwanyoung Lee et al. · 0 citations
Open access Oct 2026

Open-domain image editing with FLUX and structural causal modeling

Open-domain image editing requires accurate semantic modification while preserving identity, structure, and non-target visual content. Existing diffusion- and rectified-flow-based editors often remain affected by attribute coupling, spurious correlations, and unintended changes to semantically related regions. This stu...

Jing Yang, Fei Zhou · 0 citations
Preprint Oct 2026

TGSR: Text-Grounded Diffusion for Faithful Image Super-Resolution

Diffusion models have achieved remarkable success in image super-resolution (SR) by leveraging strong generative priors to enhance perceptual quality. However, for textual content, these priors often produce text hallucinations: non-existent or distorted text, which alter scene semantics and compromise model reliabilit...

Zhi-Ming Hu, Ran Zhang, A. Ye et al. · 1 citation
Nov 2026

SENTRY-CD: a synthetic data generation framework for military scene change detection

Matched before and after defence imagery is difficult to obtain, and many public remote sensing benchmarks focus on large, long-term structural changes rather than short-term changes involving small mobile targets. We present SENTRY-CD, a synthetic data generation framework combining open-vocabulary grounding, promptab...

Sukriti Bhattacharya, O. Ermis, Cecilia Pisano et al. · 0 citations
Sep 2026

RefCompose: Multi-Reference Image Generation via LoRA-Conditioned Diffusion

On the Dense Layout protocol, RefCompose consistently outperforms layout based and state of the art multi reference baselines on color, texture, shape, spatial accuracy, and identity/content preservation at higher reference counts, all with constant inference memory, making it a practical building block for multi subje...

Sai Sri Teja Kuppa, Parth Shinde, S. PriyadharsanBalaji et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Oct 28, 2026

Making machines that can listen

Professor Paris Smaragdis develops computers with superhuman hearing for applications like speech recognition, music understanding, and environmental monitoring.

Microsoft Research Blog Aug 11, 2026

Introducing CARE-X: Towards Clinically Useful Radiology VLMs with Auxiliary Supervision, Reward-Aligned Learning, and Tool-Augmented Measurement

Radiology AI is evolving beyond report generation. CARE-X explores a unified approach that combines flexible reasoning, calibrated predictions, and measurement-based tools for chest X-ray interpretation. The post Introducing CARE-X: Towards Clinically Useful Radiology VLMs with Auxiliary Supervision, Reward-Aligned Learning, and Tool-Augmented Measurement appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.