Evaluating Large Language Models with a Unified Prompt-Based Pipeline for Turkish News Summarization and Tag Extraction

Abstract

As access to information grows ever faster, the acceleration of news flow demands the automatic and accurate analysis of texts. This study aims to measure the multitasking performance of summarizing and tag extraction tasks in Turkish news texts using a prompt-based unified pipeline with a small dataset, employing different prompt techniques. Summarization and tag extraction tasks were performed comparatively using zero-shot and few-shot learning approaches with commercial models (GPT-40 and Claude 4.5 Sonnet) via official API providers and open-source models (Qwen3-32B and Llama 3.3-70B-Instruct) on a local server using a vLLM infrastructure. Experiments were conducted with deterministic parameters and standardized system prompts. Experimental results show that the Claude 4.5 Sonnet model exhibited the highest performance in the 3-shot scenario with a BERTScore of 0.676 and an LLM-as-a-judge score of 7.67. Furthermore, it has been observed that switching to a few-shot strategy resulted in up to a 35% increase in tagging success. The Llama 3.3 model was found to be competitive with commercial models in the tagging task with an F1 score of 0.287. These results also validate the proposed unified pipeline method in terms of accuracy and semantic similarity using BertScore, F1 score, and the LLM-as-a-judge method, which simulates human judgment. The proposed framework is expected to contribute to the literature by offering a reliable automation solution that increases efficiency with small data requirements in Turkish news processing processes.