MoganBert-TR, a 149M-parameter Turkish encoder foundation model trained from scratch on a language-specifically filtered corpus, together with an embedding model derived from it (MoganBert-Embed) that ranks first among student models on the MTEB(Turkish) overall average and outperforms all compared Turkish tokenizers on compression and fertility across two independent test sets.
Abstract
Turkish encoder models have adopted modern architectures while leaving the pretraining objective fixed at masked language modelling. This paper introduces MoganBert-TR, a 149M-parameter Turkish encoder foundation model trained from scratch on a language-specifically filtered corpus, together with an embedding model derived from it (MoganBert-Embed). MoganBert-TR is trained over 237.3B tokens with a two-stage CLM-to-MLM curriculum: causal language modelling first, masked language modelling for the remainder, with the transition made inside the stable phase of a WSD schedule. In a controlled ablation under an equal step budget, this design outperforms pure MLM by 2.7-3.7x on Turkish MS MARCO retrieval; the measured mechanism is embedding geometry, where a single direction absorbs 28.1% of the variance under pure MLM against 11.9% under the curriculum. Long-context extension and learning-rate decay are then split into two branches after a shared prefix: running the final portion of decay at 1024 context improves the TrGLUE average by 0.49 +/- 0.26 points across five paired seeds (p = 0.013) and beats a model-soup alternative by 0.75 points at ~4.3% additional cost. MoganBert-TR attains 78.41 on TrGLUE, the best among the Turkish ModernBERT models compared, and 77.73 on TabiBench, where it leads two of the eight categories with the largest margin on code retrieval (+3.62 points over TabiBERT). MoganBert-Embed, produced through teacher distillation and multi-signal contrastive fine-tuning, ranks first among student models on the MTEB(Turkish) overall average with 68.30 and reaches 99.5% of its 7.57B-parameter teacher's score with a 51x smaller backbone. The accompanying 50,048-token tokenizer outperforms all compared Turkish tokenizers on compression and fertility across two independent test sets. Weights, tokenizer, embedding model and evaluation code: https://huggingface.co/moganai
We previously reported a ModernBERT encoder trained from scratch for Turkish (MoganBERT-TR) and a single-vector embedding model built on top of it (MoganBERT-embed). This work introduces the third model in that lineage: MoganColBERT-TR, a multi-vector retrieval model that, instead of compressing a query or a document into a single vector, represents it at the token level through a 768->128 projection and scores it with MaxSim late interaction. The model is not trained from scratch: the embedding model's encoder is taken as the starting point and adapted to the ColBERT objective with a single-epoch distillation phase. Training data is produced from two sources - title-to-passage pairs carved out of our own pretraining corpus in the character domain and at sentence boundaries, and two Turkish question-based retrieval sets - and is distilled from the soft scores of a cross-encoder teacher (bge-reranker-v2-m3) over one positive and seven mined negatives. We show that in hard negative mining, rank-based skipping alone is insufficient and must be combined with a group mask and a cosine ceiling. Evaluation is carried out with the official pipeline of TurkColBERT, a benchmark built for Turkish late-interaction retrieval (PLAID index, exact MaxSim), on five Turkish BEIR datasets; none of them appears in our training pool, so all five results are clean zero-shot. With 148.9M parameters, MoganColBERT-TR reaches an overall score of 37.36 (35.53 nDCG@100, 31.81 nDCG@10) averaged over the five datasets and finishes second among the five models compared: it outperforms the twice-as-large ColmmBERT-base-TR on four of five datasets and by +3.05 overall, and the benchmark's largest model by +12.30. The gap to the leading model (mLateOn) is concentrated on ArguAna-TR, the dataset with by far the longest queries.
Furkan Yilmaz, Habibe Aleyna Tasdemir, Muhammed Faruk Gozay· 0 citations
Control studies on model depth, learning-rate magnitude and scheduling, the interaction between learning-rate decay and data quality, and the effect of including instruction data during pre-training are reported, and an unexplained surge in benchmark performance midway through the constant-learning-rate phase is documented.
Lusheng Zhang, Shien He, Tianxing Yan et al.· 0 citations
B1ade, an efficient RAG architecture comprising two purpose-built components: a compact embedding model and a purpose-built SLM shows that strategic model composition and reward design suffice for resource-efficient RAG, without large-scale pretraining.
S. Subramanian, M. Gungor, Vikram Elango· 0 citations
A bidirectional Mamba encoder pretrained via masked language modeling on a corpus combining Arabic Wikipedia and CulturaX text is introduced, trained end-to-end on four consumer-grade NVIDIA RTX 2080Ti GPUs (11GB) over approximately ten days.
Ahmed Amine Aliane, H. Aliane, N. Semmar· 0 citations
This paper proposes a general optimization framework that combines a vocabulary pruning method with a targeted fine-tuning protocol for MNMT models, and reduces the vocabulary size from over 128,000 to approximately 10,000 tokens, enabling a 60% memory saving without any loss in performance.
Ahmed Amine Aliane, N. Semmar, H. Aliane· 0 citations
Pre-trained contextual language models have demonstrated excellent performance in training with data from new languages and tasks. For this purpose, additional pre-training is necessary, since the lack of vocabulary of the language always tends to degrade the results. In this article, we present a procedure to deal with and treat unknown languages or languages without available resources such as Kicongo, a Bantu matrix language, commonly spoken in the northern regions of Angola, with greater incidence in rural regions and in the countries of central Africa (Democratic Republic of Congo, Republic of Congo and Gabon) and throughout the world. With the extension of Natural Language Processing (NLP) models based on the Bidirectional Encoder Representations from Transformers (BERT) architecture (BERT, RoBERTa and DistilBERT), the performance evaluation of the models was carried out using metrics such as accuracy, precision, recall and f1-score, achieving a performance rate higher than 97%.
Henriques Fernando, Rodrigo Florencio da Silva· Revista Brasileira de Comput...· 0 citations