Aug 2026· Scientometrics· Vol 131, pp. 6773 - 6788· 0 citations· 28 references
TL;DR
LLMAspectPrediction, a novel framework designed to predict fine-grained aspect scores for academic papers, is proposed, which assists reviewers by providing consistent, criteria-driven assessments and offers authors actionable feedback aligned with peer review standards.
Abstract
Peer review is a fundamental process in scholarly publishing, wherein reviewers assess and score various aspects of a manuscript (e.g., novelty, clarity, and significance) based on established evaluation criteria. However, this process demands substantial time and effort, and remains inherently susceptible to human bias and inconsistency. To address this issue, we propose LLMAspectPrediction, a novel framework designed to predict fine-grained aspect scores for academic papers. This framework assists reviewers by providing consistent, criteria-driven assessments and offers authors actionable feedback aligned with peer review standards. The proposed methodology comprises three sequential stages. First, raw manuscript texts are preprocessed, and a vector database is utilized to retrieve topically similar papers from an external corpus to enrich the contextual input. Second, prompt templates anchored in standard peer review rubrics guide a Large Language Model (LLM) in generating evaluations for specific aspects. Finally, these LLM-generated evaluations serve as weak supervision signals to fine-tune a pre-trained model for robust score prediction. Experimental results demonstrate that our approach achieves state-of-the-art performance, and ablation studies confirm the critical contribution of each component to the overall model efficacy.
Research on large language model (LLM)-based automated assessment (AA) has expanded rapidly. Nevertheless, the literature remains fragmented across contributions, models, implementation configurations, datasets, and evaluation metrics, complicating efforts to identify approaches suitable for personalized learning. This...
H. D. Septama, A. Permanasari, R. Ferdiana· IEEE Access· 0 citations
An LLM-based framework is proposed that leverages full-text key-insight extraction to enhance literature classification and implemented a confidence-weighted voting (CWV) mechanism using multiple LLMs to improve robustness.
Zihan Song, Shan Huang, Ngeemasara Thapa et al.· 0 citations
Using the ICLR 2025 review process, this study compares 2401 human reviews with 7203 reviews produced in separate, context-isolated API runs using Claude Sonnet 4.5, GPT-5.2 Thinking, and Gemini 3 Pro Preview across decision agreement, review-text characteristics, inter-model consistency, and human–AI aggregation.
Zhi-He Yang, Xiao-Yue Zhou, Hong-Sa Wang et al.· Publications· 0 citations
This study introduces TrustReviewer, an open-source LLM-based system for generating peer reviews of AI and machine learning papers and describes a concrete risk of recursive reviewer training and provides practical interventions for preserving judgment diversity and improving recommendation alignment in AI-assisted sci...
Results highlight consistent trade-offs across representations and matchers: two-stage re-ranking improves early-rank performance; keyword-aware Sentence-BERT increases top-3 concentration; and KG edge overlap is competitive on full abstracts, while some graph variants substantially concentrate reviewer workloads.
Farid Bagheri, Davide Buscaldi, D. Recupero· IEEE Access· 0 citations
The results show that BART achieves the best performance with an ROUGE-2 F1-score of 0.40664, while T5 demonstrates superior grammatical acceptability, achieving 93.36%, but BART achieves a very near performance to T5.
Emad Nabil· Islamic University Journal o...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.