This work systematically compares LLMaJ methods with and without decomposition on multiple NLG datasets and finds that, when human labels are available, LLMaJ without using task decomposition can perform comparably to human annotators.
Abstract
The LLM-as-a-judge (LLMaJ) framework has emerged as a promising solution for cheap, reproducible, reference-free Natural Language Generation (NLG) evaluation. Prior work seeks to improve LLMaJ by decomposing evaluation tasks into simpler sub-tasks. In this work, we systematically compare LLMaJ methods with and without decomposition on multiple NLG datasets. We find no evidence that LLMaJ with task decomposition leads to performance gains over a fair baseline that does not use decomposition. Instead, we find that previously reported performance gains in decomposition-based LLMaJ stem from using human labels as training data, and not task decomposition itself. Also, we find that, when human labels are available, LLMaJ without using task decomposition can perform comparably to human annotators.
A novel knowledge graph-based evaluation framework is proposed introducing Semantic Structural Similarity for KGs (S3KG), a hybrid similarity measure integrating structural and semantic similarity into a continuous evaluation score, alongside a diagnostic framework for categorizing reasoning errors.
Subavarshana Arumugam, Mamta Nallaretnam, K. Wickramasinghe et al.· 0 citations
This paper introduces a fully graph-based pipeline where the classifier never directly processes the input text, and reveals that graphs and text are complementary: combining both modalities achieves 92.1% accuracy on SNLI.
Younes Boufouss, Luc Pommeret, Thomas Gerald et al.· 0 citations
This work introduces SteerWrite, a training-free framework designed for personalized co-writing that effectively adapts the base model to specialized domains without gradient updates, with specific designs tailored to small datasets.
Wenhao Mao, Chengbin Hou, Wei-Xiao Wang et al.· 0 citations
LigBench is proposed, an automated evaluation benchmark that enables fine-grained and reliable evaluation of AI research ideas, consistently applicable across different generation distributions, and PAIR-IQ, a dataset tailored for training pairwise idea judgment models and serving as an auxiliary reference to support m...
Chen-Run Wang, Mingxuan Zhu, Tian-Cheng Huang et al.· 1 citation
This article proposes a new method for merging fine-tuned neural language models for the same task, derived from the same encoder-based model, based on Procrustes analysis, and shows that this method can improve upon existing merging methods in most tested configurations.
Olivier Ferret· International Conference on...· 0 citations
An iterative framework that automates the extraction of interpretable, schema-bound categorical features from unstructured text for tabular prediction models that guarantees instance-level interpretability and provides a fully transparent, semantic audit trail for every prediction.
A new computational approach reveals how cells communicate to assemble into tissues. This could accelerate stem cell engineering, improve disease models, and inform therapies.
Microsoft Research Blog· microsoft.comOct 15, 2026
Zero-knowledge proofs can verify computing results without rerunning workloads or exposing sensitive data. Magnetar helps bring them into the mainstream with support for standard programming languages, major speed gains, and quantum-resistant security. The post Proof-based general-purpose computing with Magnetar appeared first on Microsoft Research.
MIT News · Artificial Intelligence· news.mit.eduSep 24, 2026
