Skip to content

Learning to Learn from Context: Synthetic Training from Perturbed Public Documents

Sep 2026 · 0 citations · 52 references
Computer Science

TL;DR

This work attempts to make use of high-quality public documents with small perturbations and empirically finds that LLMs can successfully generate context-dependent reasoning traces and answers, which are then used to train a student model that substantially improves the performance on CL-bench.

Abstract

Real-world tasks often require large language models (LLMs) to learn from complex task-specific context rather than pretrained parametric knowledge. This capability remains a weakness of LLMs, while human annotation for such task contexts is expensive and difficult to scale. Public high-quality documents are an abundant alternative, but much of the public web has already been consumed during pretraining: training on such documents naively would reward memorization rather than context learning. In this work, we attempt to make use of high-quality public documents with small perturbations and empirically find that LLMs can successfully generate context-dependent reasoning traces and answers, which are then used to train a student model. Specifically, we construct a synthesis pipeline that (i) rewrites source documents to reduce memorization risk, (ii) generates questions and rubrics that require reasoning over the document, (iii) answers the questions with the document as context, and (iv) admits only samples that genuinely depend on the document. Without any human annotators, our pipeline generates about 10k samples from 3.5k documents, and the resulting student model substantially improves the performance on CL-bench. SFT raises a Qwen3.6-35B-A3B student from 13.7% to 22.8%, and a subsequent rubric-reward RL stage reaches 24.6%, on CL-bench comparable with a frontier model of over a trillion parameters, Qwen3.8-2.4T (23.9%). We also observe a broad transfer of improvements to long-context understanding, instruction following, and reasoning, while code generation and knowledge remain mostly flat. We hope this work provides a reproducible and scalable way to improve the ability of LLMs to learn from context, and to facilitate further research on context-grounded reasoning.

View source

Similar papers

#natural language process... Preprint Sep 2026

LLMs Learn Better In-Context from Rules than from Examples

It is found that models generally learn more reliably from rules than from examples alone, and additional examples on top of rules or simply scaling up the number of examples do not lead to consistent and significant gains.

Xiang Fu, Seungmin Cho, Yukyung Lee et al. · 0 citations
Sep 2026

Making Large Vision–Language Models Better Few-Shot Learners

Few-shot classification (FSC) aims to emulate the human ability to rapidly learn new concepts from a handful of examples. Large Vision-Language Models (LVLMs), with their rich prior knowledge and powerful visio-linguistic understanding capabilities, are emerging as a highly promising paradigm for FSC. This paper invest...

Chuan-Yi Zhang, Fan Liu, Yi Xu et al. · 0 citations
Preprint Aug 2026

What to Forget in Unlearning? Forget Set Curation for Language Models

This work introduces CleanSlate, a benchmark for verbatim output suppression over songs and books, with model-specific extraction profiles, content-grounded QA, and capability-retention evaluations, and shows that practical unlearning is not only an optimization problem once a forget set is given, but also what can be...

Animesh Jha, Arpandeep Khatua, Youssef Allouah et al. · 0 citations
Preprint Aug 2026

Training a Knowledge Base: Supervised Structure Learning for Agent-Curated Document Stores

This work evaluates the curated store with a test set, on two contamination-free benchmarks: KBGym, a fictional-universe generator the authors release, and PhantomWiki, a fictional-universe generator they release, and PhantomWiki, a fictional-universe generator they release.

Yu Pan, Hongfeng Yu · 0 citations
Preprint Sep 2026

AudioICL-Bench: A Benchmark for Large Audio Language Model In-Context Learning

AudioICL-Bench is introduced, a diagnostic benchmark whose per-episode rules are resampled so that no correct answer is recoverable from prior knowledge, and its nine tasks are organized along two axes that separate what must be learned from demonstrations from what must be perceived in the signal, enabling failures to...

Jia-Hung Chen, Yi-Cheng Lin, Kai-Wei Chang et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.