SAGA: Score-Weighted Adaptive Generation Alignment for Low-Resource Nordic Language Models

Abstract

Language models for low-resource languages still make frequent grammatical errors. Alignment methods such as RLHF and DPO can correct them. However, they require human judgements, which are scarce for languages such as Icelandic. Automatic rewards offer an alternative, yet models can learn to exploit them without improving (reward hacking). We present SAGA (Score-weighted Adaptive Generation Alignment), which builds preference pairs using dependency parsers. Such parsers already exist for many low-resource languages. SAGA's reward combines parse quality with lexical diversity. It keeps only pairs whose reward gap reaches a threshold and trains on them with DPO. On Danish, Icelandic and Norwegian Bokm{\aa}l, SAGA raises reward-parser success by 18 to 28 points. It improves all four model families we test, although one Icelandic run is flagged for reward hacking. An independent parser confirms the gains, and native speakers prefer SAGA outputs in Danish (69%) and Icelandic (80%). Online RL methods exploit the Icelandic reward. On the same base model, SAGA scores higher on both parsers. Existing linguistic resources, combined with safeguards against reward hacking, can therefore stand in for human judgements. This makes grammatical alignment possible for the many languages that have parsers but no such data.