TL;DR
A data-centric analysis of semantic knowledge acquisition in word embeddings, focusing on word analogy and semantic similarity shows that, for relational semantics, training-data quality outweighs quantity, and that simple proxy models remain a practical, interpretable tool for efficient data selection.
Current advances in neural network models have improved state-of-the-art performance in natural language processing tasks such as named-entity recognition, sentiment analysis, and machine translation. In particular, neural language models are applied to encode information in word embeddings. These approaches are generally trained on large corpora using semi-supervised learning. Word embeddings encode the syntactic and semantic properties of words as dense vectors. In agglutinative languages such as Turkish, Finnish, and Hungarian, word-embedding construction is challenging because extensive suffixation and polysemy can cause information loss. To overcome these limitations, character n-grams are often preferred for embedding representations. Nevertheless, character n-grams do not guarantee the capture of information in long word sequences. In this study, a method that partitions word sequences according to frequent patterns within a given context is proposed for training a neural language model. In this respect, likelihood- and ranking-based inference are combined with n-gram and syllable partitioning for word-embedding generation from a text corpus. The proposed approach provides a language-agnostic, context-sensitive segmentation mechanism that can complement language processing methods such as lemmatization, morphological analysis, and stemming. For embedding generation, the SkipGram and FastText models are used, and the effects of word partitioning are evaluated using analogy, named-entity recognition, POS tagging, sentiment analysis, and morphological disambiguation datasets for Turkish. The results indicate task-dependent and generally limited improvements over traditional token-based word-embedding extraction. In particular, skip n-gram partitioning produces a substantial improvement over partitioning based on frequent-ngrams, sentencepiece-bpe, sentence-unigram and morfessor. No consistent relationship was observed across tasks between performance and either graph density or the average number of distinct n-grams per sentence.
LMEnt is released to support studies of knowledge in LMs, including knowledge representations, plasticity, editing, attribution, hallucinations, and learning dynamics, finding that entity co-occurrence and mention forms—which are difficult to study with existing tools—affect learning trends.
Protein language models (PLMs) have transferred the latest advances from natural language processing to computational biology. These models, trained on large corpora of protein sequence data, are widely used to translate amino acid sequences into latent-space embeddings, ready for use in diverse downstream tasks (DTs). By a common consensus, embeddings from the model's last layer are used, and the model's internal behavior remains poorly understood. We analyzed 13 PLMs across 15 DTs from 11 datasets to investigate the informativeness of embeddings created in intermediate PLM layers. We trained probe models on embeddings from each layer, compared their performance, and computed characteristics of the latent spaces they span to estimate the information they contain, and found that the last layers of PLMs rarely contained embeddings that led to the best results on downstream tasks. Furthermore, we identified a connection between DTs and the distribution across PLMs'layers of the relevant information to predict that task. For example, similarity between the pre-training objective and the objective of predicting properties of individual residues leads to a steady increase in understanding of such tasks across the layers of PLMs. On the other hand, for whole-protein tasks, we observe that the dataset, rather than the task itself, defines PLMs'ability to perform well on a DT. Embeddings from shallow layers of PLMs perform better for datasets that contain deep mutational scan (DMS) data, while datasets containing diverse natural proteins find most useful embeddings in the models'deeper layers. Additionally, we discover that the performance of PLMs drops significantly when tasks are introduced for artificial proteins.
The findings suggest that vocabulary development can be understood through the non-trivial interplay between activation dynamics and some degree of constraints regulating the visiting of lexical categories in complex networks.