AURA, a unified multimodal framework for conversational music editing, uses a multimodal large language model to interpret the complete dialogue history, an optional image, and reference audio, distilling the editing intent into compact concept tokens, enabling precise edits while preserving unaffected content.
Abstract
Instruction-guided music editors typically process each request independently, limiting their ability to support workflows in which users progressively refine a track. We introduce AURA, a unified multimodal framework for conversational music editing. AURA uses a multimodal large language model to interpret the complete dialogue history, an optional image, and reference audio, distilling the editing intent into compact concept tokens. A concept-to-audio module injects these tokens and frame-aligned reference features into a frozen MusicGen backbone, enabling precise edits while preserving unaffected content. AURA optimizes only 91M parameters while retaining 1.9B frozen backbone parameters. Experiments on Slakh2100 and MoisesDB demonstrate substantial improvements in edit correctness and content preservation over existing instruction-guided methods, including a 4-5 times reduction in FAD for out-of-domain addition and removal.
Most music-generation systems are still framed and evaluated primarily as producers of complete outputs, whereas composition often proceeds through successive revisions to a shared musical artifact. This paper studies a different use of a general-purpose instruction-following large language model: not as a one-shot mus...
A. Fernandes, Jean-Pierre Briot, S. Barbosa et al.· 0 citations
Transcribing piano performances into editable notation remains difficult without dedicated symbolic instruments, controlled recording setups, or substantial manual correction. This paper presents an early prototype for transforming piano performance videos captured from a keyboard-oriented viewpoint into draft MusicXML...
Silvia Di Giulio, Luigi Troiano, G. Tortora· Companion Publication of the...· 0 citations
Video editing spans diverse editing paradigms, yet achieving high-quality instruction-guided and subject-guided editing within a single unified framework remains challenging. We introduce EditVid, a training-free framework combining sparse causal memory for local coherence, correspondence-based post-attention token inj...
A. Juvekar, O. Susladkar, K. A. Nguyen et al.· 0 citations
Text-to-music systems produce increasingly convincing audio, yet evaluation reveals little about whether the result matches user intent. A global text-audio relevance score can overlook the implicit intent in underspecified prompts and mask failures in specific requirements, such as instrumentation, structure, rhythm,...
Ze-Kai Liu, Zhi-Lin Wang, Xu-Zheng He et al.· 0 citations
Spatial audio editing modifies an existing soundfield according to a user's instruction while preserving the rest of the scene. Unlike conventional audio editing, it must reason jointly about audio events, spatial information, dynamic changes, and environmental information in first-order Ambisonic (FOA) waveforms. Exis...
Ke Lei, Chenyuhao Wen, Yu Zhang et al.· 0 citations
We present AuraLuxMuse, a novel system for automated aesthetic stage lighting design that integrates expert knowledge, representation learning, and preference-adaptive modeling. Lighting design in live performance settings requires the seamless translation of musical features into dynamic lighting behaviors. However, t...
Jun-Yu Deng, Jia-Le Cao, Meng-Tian Li et al.· 0 citations
