AI for Creative Visual Content Generation Editing and Understanding

JarvisHub: An Open Harness for Canvas-Native Multimodal Creative Agents

Creative AI is moving from single-step asset generation toward long-horizon multimodal production. Although recent generative models can synthesize high-quality images, videos, audio clips, UI elements, storyboards, slides, and other creative assets, real-world creative work requires more than isolated prompt-output interactions. It involves references, drafts, alternatives, edits, failed attempts, version relations, tool actions, evaluation signals, and human feedback, which together form an evolving project state. Existing prompt-based, chat-based, and node-based generation systems only partially support this state, as they often discard intermediate context, rely on linear conversations, or require manually specified workflows. Recent commercial systems indicate a shift toward agent-assisted creative production, but their closed architectures make it difficult to study how agents represent context, choose tools, revise artifacts, recover from failures, and maintain consistency over time. To address this gap, we introduce JarvisHub, a canvas-native creative agent harness for long-horizon multimodal creation. JarvisHub treats an editable canvas as the user workspace, the agent's external memory, action space, and shared project state, representing multimodal artifacts, dependencies, versions, and feedback as typed canvas nodes and links. Through a three-layer architecture of canvas state, protocol bridge, and agent runtime, JarvisHub enables agents to act within an inspectable and editable creative state. This design moves creative agents beyond isolated tool use toward sustained, human-steerable creative automation, where agents can progressively plan, generate, revise, and organize multimodal projects while users remain able to inspect, guide, and intervene throughout the process.

Trajectory as Creative Intent: Rethinking Drag-Based Interaction in Generative Visual Editing

ArtChart: Faithful Artistic Chart Generation with Integrated Text Rendering

Artistic charts combine data visualization with expressive marks, textures, and typography, but they are difficult for image generators: an output is useful only when its stylization preserves chart geometry, exact in-image text, and the semantic binding between labels and marks. We introduce ArtChart, a framework for faithful artistic chart generation with integrated text rendering. Given a structured chart specification and an artistic prompt, ArtChart first renders a text-free grayscale layout that encodes the target chart geometry, then trains a chart-specific control module to preserve mathematical structure. To address the remaining text and layout errors, we further refine the generation policy through GRPO-based reinforcement learning with OCR-based text rewards, VLM-based layout rewards, and aesthetic rewards. A multi-expert distillation stage reconciles these objectives by distilling single-reward experts into one balanced generation policy. We also construct ArtChart-Bench, a bilingual 2K-prompt benchmark covering four chart types, controlled value distributions, diverse label/value formats, and 15 artistic styles, together with ArtChart-Eval, a six-axis evaluation protocol measuring mathematical logic, text accuracy, text layout, aesthetics, instruction following, and readability. Experiments on ArtChart-Bench show that ArtChart consistently outperforms prompt-only, image-editing, and generic ControlNet baselines, with the largest gains on mathematical fidelity and label-layout binding while maintaining competitive visual quality. These results suggest that artistic chart generation should be evaluated as reliable visual communication rather than as generic stylized image synthesis.

Ctrl + Create: Empowering Creative Control in AI-Driven Rapid Level Design

Level design is one of the most labor-intensive processes in video game development – especially because communicating a creative vision across designers, artists, writers, and gameplay engineers requires extensive iterative refinement. Traditionally, this involves rounds of white boxing and set dressing, which distributes labor effectively but limits rapid exploration and creative experimentation. Recent advances in AI-driven image generation offer timely opportunities to transform these workflows, but risk losing control to the model’s interpretation and capabilities. We investigate the impact of generative AI on designers’ control, expressiveness, and efficiency through a mixed-methods study (n=20), comparing drawing (full control), text-to-image (full AI), and our approach: a visualization pipeline combining generative AI with user-centered spatial control. It succeeded in enhancing visual expressiveness and sense of control over bare AI, matched manual drawing (without necessitating advanced skills), and enabled faster iteration – highlighting the potential of giving back control to creative visionaries.

Trinketry: Tracing and Recombining Visual Elements in Generative Design Exploration

Generative AI systems allow visual designers to rapidly produce many image alternatives, but preserving, revisiting, and building on promising partial results across iterations remains challenging. In practice, generated images are rarely valuable as complete wholes; instead, designers often reuse fragments, revisit earlier materials, and combine partial ideas into new directions. We present Trinketry, an element-centered visual exploration system built around trinkets: reusable visual and semantic elements extracted from prior generations. Trinketry allows users to extract image regions, whole images, text labels, and composite elements; recombine them in intention trays to generate new variations; trace the provenance of specific elements across the exploration process; and retrieve semantically related prior outcomes. By foregrounding extraction, recombination, tracing, and retrieval, this demo presents an alternative interaction model for AI-assisted visual exploration—one that supports traceable reuse, intentional iteration, and reflective design exploration.

Synthetic Gestures: An Evolutionary Sketching Machine

There is growing interest in using pretrained machine learning models for non-photorealistic rendering and sketch synthesis. However, existing approaches typically rely on differentiable rendering, which can be constraining for artists working with algorithmic processes and mechanical plotters. This pictorial introduces a gradient-free, artist-centred method for synthesising sketches from text prompts, designed for use with a plotter. Drawing gestures are parameterised as implicit neural representations and optimised using an evolution strategy, with semantic guidance from CLIP, a vision-language model. The system runs locally on laptops and provides real-time visual feedback for artistic iteration. Its gradient-free design enables flexible, non-differentiable rendering pipelines implemented in the browser using familiar JavaScript graphics toolkits. An early version was exhibited in 2025 during a group exhibition in Tokyo, functioning both as an artwork and a research-through-art investigation into how machine learning systems might better align with artists’ practices.