Similarity-Based Prompting of Instruction-Tuned Models for English and Translated Indonesian News Summarization

Abstract

Retrieving semantically similar demonstrations is widely reported to improve in-context learning, but that evidence comes largely from classification and short-form generation, where demonstrations fit comfortably in context. We ask whether the benefit transfers to long-document news summarization, and whether it survives instruction tuning. BART-Large, T5-Large, and FLAN-T5-Large are evaluated on 1000 test documents from CNN/DailyMail and 1000 from IndoSum, the latter machine-translated into English so that language is held constant. For FLAN-T5-Large we compare zero-shot prompting against prompts containing k=2 and k=5 demonstrations retrieved from the training pool by TF-IDF cosine similarity, scoring with ROUGE-1/2/L and BERTScore. Under natural-language prompting, FLAN-T5-Large shows a large performance advantage over the two non-instruction-tuned baselines, but the design does not isolate instruction tuning as its cause. Within FLAN-T5-Large, TF-IDF retrieval with heavily truncated demonstrations changes average ROUGE-1 by only 0.15 points on CNN/DailyMail (31.29 to 31.44) and 0.51 on IndoSum (51.48 to 51.99), while BERTScore F1 is flat or marginally lower, at an inference-time cost of up to 14.9%. Because no inferential significance test is reported, these small differences are treated as descriptive rather than as evidence of a statistically established zero effect. The weak average gains are consistent with three entangled constraints: demonstrations were truncated to 25 words, IndoSum neighbours had low lexical similarity, and all prompts exceeded FLAN-T5's 512-token pre-training length. We further show that the substantially higher absolute scores on IndoSum reflect extractive references compounded by translating source and reference through the same machine translation (MT) system, and caution against treating translated IndoSum as equivalent to native-English evaluation or comparing ROUGE across these corpora.