Optimized architecture based on multilayer contextual embeddings for evaluating semantic similarity of texts in the Uzbek language

Abstract

Semantic text similarity is an especially hard task that can be performed in the Uzbek language because of its rich morphological and the absence of annotated data. The given paper introduces a very effective Bidirectional Encoder design using the monolingual model named “Bidirectional Encoder Representation from Transformers for Uzbek language” and optimized to provide scalable semantic search. A combination of knowledge distillation and metric learning are applied in a semi-supervised approach that is employed by the model. Triplet Loss with Hard Negative Mining is used to enhance the discriminative ability of the vector space. One such innovation is supporting the Matryoshka Representation Learning, which allows the model to produce dynamically truncated dimensions of embeddings. The architecture proposed wound up with a Spearman correlation of 0.835 within the Uzbek test set, topping the state-of-the-art results. Furthermore, Matryoshka Representation Learning achieves 6-fold compression of vectors with a negligible accuracy drop (ρ=0.816), having high-computation efficiency to implement Natural Language Processing systems in low-resource systems.