Semantic and Syntactic Preserving Parallel Corpus Augmentation for Neural Machine Translation

Abstract

In the domain of Natural Language Processing (NLP) Parallel text corpora are crucial for training machine translation systems and multilingual NLP applications. They provide the necessary context for learning translation models (e.g., statistical MT or neural MT). In addition, they foster semantic, syntactic, and structural equivalences between the pair of languages (bitext). One of the major difficulties in MT using parallel corpora is the insufficiency of corpora size, in terms of the number of words and the span of vocabulary. Therefore, to tackle this difficulty in this paper, we propose a novel approach to the augmentation of parallel text corpora. The approach allows for improving the quality of translation and the capability to expand the original corpora manyfold in size. The proposed approach is economical, as it does not require any additional monolingual corpora. We use the Multi-Lingual Masked Language Model (XLM-R) to mask and predict alternative words in context. Here, we demonstrate our approach on a bitext corpus of just one sentence pair, that is, a single English sentence and a single Hindi sentence. Applying our methodology to this pair, where the first language $L 1$ is English, $50(N 1)$ new sentences were generated. For the second language, Hindi $L 2,60(N 2)$ new sentences were generated. Thus, the total number of sentences available for pairing is $N 1 \times N 2$. That is, now we have 3000 possible pairings. Next, we use Sentence Embeddings (LaBSE) to check and select sentence pairs that are likely to be translations of each other. We cross-check our method using metrics for MT Quality Estimation (TransQuest). Of these 3000 pairs of sentences, only 200 achieved a LaBSE_score $\geq 0.80$. This score indicates reasonably good corpus quality based on syntactic and semantic evaluation. We believe that the approach stated here can greatly alleviate the data scarcity problem for translation-related language pairs for which a reasonable seed corpus is available. In future work, we aim to address the alignment issue. It will improve the quality and quantity of the generated parallel corpus to produce higherquality sentence pairs, which will ultimately benefit NMT System.