Skip to content
Preprint

BehaviorWorldGen: Closing the Loop between Action Models and World Simulators via Controllable Behavior-Aware Structured World Generation

Aug 2026 · 0 citations · 62 references
Computer Science

Abstract

Modern driving action models are increasingly improved in a self-improvement loop, where a learned world simulator imagines future observations and the resulting data is fed back to refine the action model. However, the bottleneck of this loop lies in the simulators'inability to generate behaviorally plausible responses by surrounding agents, making generated data both unrealistic in interaction and imbalanced in distribution. We introduce BehaviorWorldGen, a framework that closes the loop between action models and world simulators through controllable behavior-aware structured world generation. Its core component is BehaviorFlow, a meta-action-conditioned traffic-flow model that injects interpretable behavior controls and jointly generates multi-agent rollouts. BehaviorFlow realizes the specified agent behaviors while allowing surrounding vehicles to respond to the ego and to one another. The resulting rollouts are rendered by a world simulator into realistic multi-view observations, which are paired with corrected interaction-aware trajectories for action-model refinement. Since BehaviorWorldGen uses structured trajectories as the interface between its modules, it is compatible with diverse action models and world simulators. Experiments on world generation, scene extrapolation, and policy refinement demonstrate consistent improvements, with the largest benefits concentrated on difficult interactive scenarios.

View source

Similar papers

Sep 2026

ReWAM: Reciprocal World Action Models for Interactive Autonomous Driving

In interactive scenarios, an autonomous driving system is required to generate ego actions under the influence of other agents'behaviors. Existing World Action Models (WAMs) typically model other agents as components of the world model rather than as decision-makers that fundamentally shape the action of the ego agent,...

Ben-Shan Ma, Pei Liu, Rui-Guo Zhong et al. · 0 citations
#artificial intelligence Preprint Oct 2026

AnyGoal: On World Model Generalization Beyond Expert Rollouts

AnyGoal, a benchmark of goal-shifted task variants built from widely used world-model environments, is introduced, together with an open-source library, to answer the question can a world model recombine familiar state and action factors into valid dynamics?

Heejeong Nam, Divyat Mahajan, Quentin Le Lidec et al. · 0 citations
Preprint Oct 2026

ProWorld: Multi-View World Model with Proprioceptive Feedback for Closed-Loop Policy Evaluation

Evaluating robot manipulation policies through repeated physical trials is costly. We study whether an action-conditioned world model can serve as a closed-loop test environment whose policy success rates track those on real robots. Beyond realistic video, the model must return the camera views and proprioceptive state...

Zi-Jun Wang, Tian-Yi Yan, Hao-Kun Lin et al. · 0 citations
Sep 2026

WorldLine: Action-Driven Visual Simulation for Robotic Manipulation

WorldLine is introduced, an action-driven visual simulator that decouples transferable dynamics learning from heterogeneous action grounding and makes WorldLine a scalable and efficient visual simulator for policy evaluation and embodied planning.

Shenghe Zheng, Wen-Bo Li, Ji-Yao Zhang et al. · 1 citation
#artificial intelligence Preprint Oct 2026

Mind the Execution Gap: Action-Semantic Mismatch in World-Model Control

This work introduces two lightweight execution-consistent interfaces, Future-Sequence for TD-MPC2 and Applied-Action Feedback for DreamerV3, that correct future-action timeline mismatch between imagined and executed action sequences without modifying the pretrained world models.

Sheng-Tao Wen, Xiang Chen, Yu Tian et al. · 0 citations

Dynamic Manipulation with World-Action Models via Counterfactual Planning

DPP enables real-time dynamic manipulation on a single consumer GPU without additional training on dynamic data and constructs a counterfactual observation that places a predicted target position in a familiar robot context, allowing the model to invoke an existing manipulation skill rather than generate a recovery beh...

Sunwoo Park, Won-Sang Lee, Seonghyun Jin et al. · 1 citation

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.