LigBench is proposed, an automated evaluation benchmark that enables fine-grained and reliable evaluation of AI research ideas, consistently applicable across different generation distributions, and PAIR-IQ, a dataset tailored for training pairwise idea judgment models and serving as an auxiliary reference to support more objective comparative evaluation.
Abstract
With the rapid advancement of large language models (LLMs), research idea generation has attracted increasing attention. Existing approaches enable LLMs to retrieve relevant literature and propose novel ideas for research areas. However, current evaluation practices for idea generation remain fragmented and lack objective standards, often relying on direct LLM scoring, which limits their ability to provide unified and reliable assessments across a coherent distribution of generated ideas. To address this challenge, we propose LigBench, an automated evaluation benchmark that enables fine-grained and reliable evaluation of AI research ideas, consistently applicable across different generation distributions. In addition, we introduce PAIR-IQ, a dataset tailored for training pairwise idea judgment models and serving as an auxiliary reference to support more objective comparative evaluation. Extensive experiments demonstrate that LigBench achieves stable and interpretable evaluations, significantly improving alignment with expert judgments. Furthermore, models trained on PAIR-IQ exhibit enhanced ranking accuracy and robustness, establishing a principled standard for scalable and objective research idea assessment.
Large language models (LLMs) are built on the classic Transformer architecture and have become a core driving force for the rapid development of modern artificial intelligence. This paper presents a systematic review of LLMs, elaborating on their fundamental working principles, mainstream open-source models, effective lightweight optimization methods, retrieval-augmented generation frameworks and key human-value-aligned technologies. Nowadays, LLMs have been widely applied in practice. Typical scenarios include intelligent text generation, professional knowledge-based question answering and automated code generation, delivering remarkable value to both industries and academia. However, their large-scale industrial application is still restricted by multiple challenges. The major issues involve content hallucination, poor model interpretability, excessive computing resource consumption, potential ethical risks and unsatisfactory multimodal integration capability. This paper also forecasts the future development directions of LLMs, such as lightweight deployment on edge devices, safety-focused human value alignment, in-depth cross-modal fusion and customized large models for vertical industries. Additionally, it collects a number of representative cases, which can offer solid references and practical guidance for relevant researchers and engineering practitioners to carry out further studies.
AutoDavis is introduced, a first-of-its-kind automatic and dynamic evaluation protocol that enables on-demand benchmarking of LVLMs across specific capability dimensions and shows effectiveness and reliability, offering a new paradigm for dynamic benchmarking of multimodal intelligence.
Han Bao, Yue Huang, Yanbo Wang et al.· Proceedings of the 32nd ACM...· 0 citations
CALAMITA is conceived as a rolling benchmark, enabling continuous integration of new tasks and models, and argues that this combination offers a blueprint for other languages and communities seeking inclusive and rigorous LLM evaluation practices.
Malvina Nissim, Danilo Croce, V. Patti et al.· Italian Journal of Computati...· 0 citations
A fundamental gap between generation fluency and reasoning ability is uncovered, a pronounced ''inverse scaling effect'' where larger models can underperform in domain-specific reasoning, and systemic bottlenecks across all SOTA models are identified, exposing fundamental limitations of current architectures.
Guangtao Nie, Huimu Wang, Gewei Lu et al.· Proceedings of the 32nd ACM...· 0 citations
The rapid growth of academic literature has intensified the need for effective automatic text summarization techniques capable of producing concise and informative representations of scientific documents. While extractive methods are widely used, they are limited in their ability to generate coherent and semantically rich summaries. Recent advances in Transformer-based architectures have enabled significant progress in abstractive summarization; however, their effectiveness on domain-specific datasets, such as scientific articles, remains an open challenge. In this study, we investigate the performance of three pre-trained Transformer-based models—T5, BART, and GPT-2—on the task of abstractive summarization using the CL-SciSumm 2019 dataset. A total of 19 experimental configurations are conducted to analyze the impact of generation parameters, including beam size, length penalties, and n-gram constraints, on summarization quality. The models are evaluated using ROUGE metrics, with a focus on ROUGE-2.To complement content-based evaluation, this work incorporates linguistic acceptability assessment using the Corpus of Linguistic Acceptability (CoLA), a benchmark dataset for evaluating grammatical correctness. The results show that BART achieves the best performance with an ROUGE-2 F1-score of 0.40664, while T5 demonstrates superior grammatical acceptability, achieving 93.36%, but BART achieves a very near performance to T5. Ultimately, these findings demonstrate the potential of pre-trained neural networks, particularly the BART architecture, to drive the future of complex, generative NLP applications, transforming how academic research is processed and understood.
Emad Nabil· Islamic University Journal o...· 0 citations
Large Language Models (LLMs) have been rapidly evolving lately, resulting in the need for strong, explainable models to detect the difference between human-generated and machine-generated articles. Existing approaches which are mostly based on fine-tuned transformers suffer from several drawbacks such as rapid obsolescence, paraphrasing attacks, and lack of interpretability. To improve their ability to detect, this paper proposes a novel paradigm called Human vs. LLM Identification (HLI) which introduces a Retrieval-Augmented Generation (RAG)-inspired evidence-based detection strategy alongside a fine-tuned transformer classifier. Our core model, DeBERTa-Sentinel, is built on top of a fine-tuned Microsoft DeBERTa-v3-small model, which uses a disentangled attention mechanism to better capture subtle syntactic and stylistic deviations characteristic of AI-generated text. We evaluate our framework on a balanced dataset of 43,456 text samples, curated from the OpenGPTText corpus and covering AI-generated and human-authored content across diverse domains including news, education, and creative text. The experimental results show improved performance over the selected baselines, with our framework achieving an accuracy of 97.53%, precision of 95.89%, recall of 99.34%, and ROC-AUC of 99.53%. In addition, explainability is integrated into our framework through Local Interpretable Model-agnostic Explanations (LIME) analysis, providing token-level insight into classification decisions. This study establishes a benchmark for scalable, explainable AI text detection, with implications for academic integrity, content moderation, and combating misinformation.
Ibtasam Ur Rehman, Muhammad Islam, Muhammad Yousaf Rehman et al.· Knowledge· 0 citations