Assessing Transformer Models for Abstractive Summarization of Scientific Articles

Abstract

The rapid growth of academic literature has intensified the need for effective automatic text summarization techniques capable of producing concise and informative representations of scientific documents. While extractive methods are widely used, they are limited in their ability to generate coherent and semantically rich summaries. Recent advances in Transformer-based architectures have enabled significant progress in abstractive summarization; however, their effectiveness on domain-specific datasets, such as scientific articles, remains an open challenge. In this study, we investigate the performance of three pre-trained Transformer-based models—T5, BART, and GPT-2—on the task of abstractive summarization using the CL-SciSumm 2019 dataset. A total of 19 experimental configurations are conducted to analyze the impact of generation parameters, including beam size, length penalties, and n-gram constraints, on summarization quality. The models are evaluated using ROUGE metrics, with a focus on ROUGE-2.To complement content-based evaluation, this work incorporates linguistic acceptability assessment using the Corpus of Linguistic Acceptability (CoLA), a benchmark dataset for evaluating grammatical correctness. The results show that BART achieves the best performance with an ROUGE-2 F1-score of 0.40664, while T5 demonstrates superior grammatical acceptability, achieving 93.36%, but BART achieves a very near performance to T5. Ultimately, these findings demonstrate the potential of pre-trained neural networks, particularly the BART architecture, to drive the future of complex, generative NLP applications, transforming how academic research is processed and understood.