Dictionary-Constrained Grapheme-to-Phoneme for Unsegmented Languages from LLM-Annotated Data
TL;DR
A context-aware, segmentation-agnostic neural G2P framework that models the joint segmentation-and-reading hypothesis space, scoring paths of a discriminative conditional random field (CRF) over a dictionary-derived word lattice over a dictionary-derived word lattice is presented.
Abstract
Grapheme-to-phoneme (G2P) conversion turns raw text into its phonemic form and is an essential part of both text-to-speech (TTS) and automatic speech recognition (ASR) systems. It is required to be fast, stable and context-aware. For unsegmented languages such as Japanese, G2P additionally couples word segmentation with highly context-dependent polyphone disambiguation, and the scarcity of accurately annotated data remains a bottleneck. In this paper, we present a context-aware, segmentation-agnostic neural G2P framework that models the joint segmentation-and-reading hypothesis space, scoring paths of a discriminative conditional random field (CRF) over a dictionary-derived word lattice. To tackle data scarcity, we utilize large language models (LLMs) to generate more than 2 million sentences. Experimental results demonstrate that our method substantially outperforms conventional morphological analyzer-based methods and neural sequence models. On the Joyo-Kanji-Yomi benchmark, our method reaches 99.62% target word reading accuracy, 0.32% target word phoneme error rate (PER) and 0.14% sentence PER.
