Skip to content
Review Open access

LLM aspect prediction: reviewing academic papers from different aspects with Large Language Model

Aug 2026 · Scientometrics · Vol 131, pp. 6773 - 6788 · 0 citations · 28 references

TL;DR

LLMAspectPrediction, a novel framework designed to predict fine-grained aspect scores for academic papers, is proposed, which assists reviewers by providing consistent, criteria-driven assessments and offers authors actionable feedback aligned with peer review standards.

Abstract

Peer review is a fundamental process in scholarly publishing, wherein reviewers assess and score various aspects of a manuscript (e.g., novelty, clarity, and significance) based on established evaluation criteria. However, this process demands substantial time and effort, and remains inherently susceptible to human bias and inconsistency. To address this issue, we propose LLMAspectPrediction, a novel framework designed to predict fine-grained aspect scores for academic papers. This framework assists reviewers by providing consistent, criteria-driven assessments and offers authors actionable feedback aligned with peer review standards. The proposed methodology comprises three sequential stages. First, raw manuscript texts are preprocessed, and a vector database is utilized to retrieve topically similar papers from an external corpus to enrich the contextual input. Second, prompt templates anchored in standard peer review rubrics guide a Large Language Model (LLM) in generating evaluations for specific aspects. Finally, these LLM-generated evaluations serve as weak supervision signals to fine-tune a pre-trained model for robust score prediction. Experimental results demonstrate that our approach achieves state-of-the-art performance, and ablation studies confirm the critical contribution of each component to the overall model efficacy.

Read PDF

Similar papers

Review Open access 2026

Large Language Model-Based Automated Assessment: A Systematic Review, Taxonomy, and Implications for Personalized Learning

Research on large language model (LLM)-based automated assessment (AA) has expanded rapidly. Nevertheless, the literature remains fragmented across contributions, models, implementation configurations, datasets, and evaluation metrics, complicating efforts to identify approaches suitable for personalized learning. This...

H. D. Septama, A. Permanasari, R. Ferdiana · 0 citations
Review Open access Aug 2026

Large Language Models in Peer Review: Decision Alignment, Review-Text Characteristics, and Human–AI Aggregation at ICLR 2025

Using the ICLR 2025 review process, this study compares 2401 human reviews with 7203 reviews produced in separate, context-isolated API runs using Claude Sonnet 4.5, GPT-5.2 Thinking, and Gemini 3 Pro Preview across decision agreement, review-text characteristics, inter-model consistency, and human–AI aggregation.

Zhi-He Yang, Xiao-Yue Zhou, Hong-Sa Wang et al. · 0 citations
#machine learning Review Sep 2026

When AI Reviews Train AI Reviewers: Scientific-Judgment Collapse and Mitigation

This study introduces TrustReviewer, an open-source LLM-based system for generating peer reviews of AI and machine learning papers and describes a concrete risk of recursive reviewer training and provides practical interventions for preserving judgment diversity and improving recommendation alignment in AI-assisted sci...

Sy-Tuyen Ho, Ming-Hui Liu, Fu-Rong Huang · 0 citations
Review Open access 2026

LLM-Assisted Reviewer Assignment via Auditable Expertise Matching

Results highlight consistent trade-offs across representations and matchers: two-stage re-ranking improves early-rank performance; keyword-aware Sentence-BERT increases top-3 concentration; and KG edge overlap is competitive on full abstracts, while some graph variants substantially concentrate reviewer workloads.

Farid Bagheri, Davide Buscaldi, D. Recupero · 0 citations
2026

Assessing Transformer Models for Abstractive Summarization of Scientific Articles

The results show that BART achieves the best performance with an ROUGE-2 F1-score of 0.40664, while T5 demonstrates superior grammatical acceptability, achieving 93.36%, but BART achieves a very near performance to T5.

Emad Nabil · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.