A Few Good Instances: How Much Data Does Frugal Keyphrase Generation Actually Need?
TL;DR
This work fine-tune BART-base under budgets of 1–70% of the training pool on three corpora from distinct domains into a training-free triage rule and release code, checkpoints and inferences, and distil this into a training-free triage rule and release code, checkpoints and inferences.
Abstract
Fine-tuning sequence-to-sequence models such as BART is the state of the art for keyphrase generation, at tens to hundreds of GPU hours per corpus. We ask whether data selection makes it frugal: does choosing which documents to train on buy anything over simply taking fewer of them at random? We fine-tune BART-base under budgets of 1–70% of the training pool on three corpora from distinct domains (KP20k, KPBiomed, KPTimes), comparing random sampling against frequency-weighted keyphrase coverage, gradient diversity (BADGE) and influence-based selection (LESS): 75 runs, 108 paired comparisons. The exchangeable unit is the (dataset, budget) block: within one block every difference is measured against the same RANDOM run. Over the 18 blocks, an exact sign-flip permutation test puts the two off-the-shelf strategies 0.83 points behind random sampling (95% CI [ − 1.55, −0.11], p = 0.026); LESS loses by 1.07 despite its privileged access to the validation set. The claim that survives any multiplicity correction is the interval: the best strategy’s 95% CI reaches only + 0.15 points, so informed selection cannot repay its own overhead. What pays is the budget. At matched exposure, 70% of the data reproduces full-data quality within 0.4 points on both metrics and all three corpora at 64–77% of the cost; on a redundant editorial corpus, 30% beats full-data present F1@5 at 35% of the cost and under half the exposure, while small budgets disproportionately damage absent-keyphrase generation. We distil this into a training-free triage rule and release code, checkpoints and inferences.1