Skip to content
Preprint

Rethinking Music Tokenization: A Semantic Codec toward High-Fidelity LLM Music Generation

Sep 2026 · 0 citations · 33 references
Engineering

TL;DR

MuSeC is proposed, a music semantic codec that factorizes semantic and acoustic content directly from mixed signals without source separation that preserves information required for high-fidelity reconstruction while producing more LM-friendly discrete units.

Abstract

Discrete audio tokenization has become the critical interface between raw waveforms and autoregressive modeling in recent music generation. As a result, music tokenizers must simultaneously support high-fidelity reconstruction and produce discrete sequences that remain amenable to language modeling. Existing reconstruction-oriented tokenizers often mix musical structure with fine acoustic details, producing high-entropy tokens that are hard to model. In contrast, semantics-guided alternatives are designed for speech and do not fit music well, often hurting reconstruction quality. We address these trade-offs by rethinking music tokenization around a measurable notion of music semantic content grounded in downstream Music Information Retrieval tasks. Guided by this definition, we propose MuSeC, a music semantic codec that factorizes semantic and acoustic content directly from mixed signals without source separation. MuSeC preserves information required for high-fidelity reconstruction while producing more LM-friendly discrete units. Empirically, it improves reconstruction quality and yields more predictable token sequences, providing a practical foundation toward high-fidelity LLM music generation. Demos are available at https://longwaytog0.github.io/MuSeC/.

View source

Similar papers

Preprint Oct 2026

Does Notation Matter? MusicXML Tokenisation for Symbolic Music Generation and Understanding

Symbolic music representations encode musical content at varying levels of completeness, from performance-oriented formats such as MIDI to fully notated scores such as MusicXML. Yet the impact of this representational richness on deep learning models remains poorly understood. While tokenisation has been extensively st...

Andrea Poltronieri, Matteo Spanio, Simone Chieppa et al. · 1 citation
Preprint Oct 2026

A learning-based method of automatic music transcription

This paper formulate polyphonic piano transcription as a frame-level multi-label classification problem across the standard 88 piano pitches, and proposes two architectural adaptations tailored to recurrent temporal representations: a modified additive attention module and a modified multihead attention mechanism equip...

Phu-Anh-Tai Do, Huu-Hoang Nguyen, Duc-Hoang Nguyen et al. · 0 citations
#artificial intelligence Preprint Oct 2026

AudioTETRIS: Token Elimination via Transitional Importance Signals for Efficient Audio LLMs

Large Audio Language Models (LALMs) achieve strong performance on speech-text tasks, but standard audio encoders produce hundreds of tokens for even short audio clips. Because self-attention scales quadratically in sequence length, this inflates inference-time compute substantially. Current training-free token compress...

Shreya Sajid, Anna Vettoruzzo, J. Vanschoren · 0 citations
Preprint Oct 2026

MSVAE: Rethinking Musical Semantics in Continuous Latent Spaces

Continuous latent representations underlie recent diffusion-based lyrics-to-song and text-to-music systems, yet aligning lyric conditions with these latents remains difficult: existing systems resort to generator-side machinery such as sentence-level timestamps, auxiliary alignment objectives, or semi-autoregressive de...

Guo-Bin Ma, Yue-Peng Jiang, Da-Ke Guo et al. · 0 citations
Preprint Oct 2026

High-Fidelity Song Generation with Joint Discrete and Continuous Modeling

Generating a full song from lyrics and a style description requires long-range musical planning without sacrificing vocal and instrumental detail. Low-rate discrete semantic tokens make this tractable, but quantization constrains the information available to acoustic rendering, while separately optimized stages prevent...

Zhe-Qi Dai, Zhi-Yue Wu, Rui-Bin Yuan et al. · 0 citations

YuE2: Unifying Symbolic and Audio Music Generation at Frontier Quality

Symbolic models make melody, harmony, rhythm, and form explicit but typically stop before a finished recording; audio models produce complete songs while leaving composition implicit. We introduce YuE2, which unifies symbolic and audio music generation at frontier quality through symbolic planning. A single AR-NAR Mixt...

Rui-Bin Yuan, Jia-Hao Pan, Jun-Yan Jiang et al. · 1 citation

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.