Skip to content

Learn Your Own Thoughts: Abstract Token Curriculum

Sep 2026 · 0 citations · 41 references
Computer Science Mathematics

TL;DR

This work proposes Abstract Token Curriculum (ATC), a novel curriculum learning framework that elicits effective continuous intermediate representations without direct supervision or manual scratchpad design and shows ATC's effectiveness on graph reachability and arithmetic learning tasks.

Abstract

Large Language Models (LLMs) have achieved remarkable reasoning capabilities by utilizing chain-of-thought (CoT) as a scratchpad for intermediate stages of thinking. However, CoT techniques require explicit supervision on thinking tokens, which requires rich, task-specific data. In this work, we propose Abstract Token Curriculum (ATC), a novel curriculum learning framework that elicits effective continuous intermediate representations without direct supervision or manual scratchpad design. ATC gradually increases problem complexity through a sequence of distributions, training the model to develop internal abstract ``thoughts''in the continuous representation space. This paper provides both theoretical and experimental evidence for the benefits of ATC and its advantages over previous methods for training continuous thoughts. Theoretically, we show that for learning parity functions with single-layer softmax attention using ATC, attention naturally focuses on the CoT tokens in the context that provide the ``easiest path''to predicting the next token. Experimentally, we show ATC's effectiveness on graph reachability and arithmetic learning tasks.

View source

Similar papers

#machine learning Preprint Oct 2026

A Systematic Study of Small Language Models on Abstract Reasoning Tasks

Endpoint accuracy on abstract-reasoning benchmarks does not reveal whether a language model has acquired a transferable rule or fit distribution-specific regularities. We study this distinction in small language models on the ARC-TGI benchmark, which organizes abstract grid transformations into controllable task famili...

Nur A. Zarin Nishat, Jens Lehmann, Andrei C. Aioanei et al. · 0 citations
#machine learning Preprint Aug 2026

A Model with No Head and Many Thoughts

This work introduces Soft Latent Thinking, a method that replaces the LM head during reasoning with a lightweight projector, enabling autoregressive rollout in embedding space where reasoning steps remain continuous rather than tokenized.

N. Koriagin, Yaroslav Aksenov, George Bredis et al. · 0 citations
#artificial intelligence Review Oct 2026

Curriculum Brain: Constructing Curriculum Knowledge Graphs as a Substrate for Cognitive Diagnosis

Cognitive Diagnostic Models (CDMs) identify which specific skills a student has and has not mastered, the signal a personalized learning path needs and a single aggregate score cannot give. Yet they are rarely deployed. The obstacle is their precondition: the Q-matrix, a mapping from every assessment item to the skills...

Shrideep Tamboli, C. Maddala, Eshal Minhaj · 0 citations

Ladders of Thought: A Self-Evolving Curriculum of Progressively Simplified Reasoning Traces

Evaluated on two reasoning domains, math and multi-hop reasoning, across 1-8B models from different families, LoT consistently improves over KD and converges faster than staged curricula, highlighting the value of adaptive progression.

Ming-Hui Liu, Thomas Magelinski, De-Hao Yuan et al. · 0 citations

Not All Thinking is Created Equal: Latent Reasoning Discovers a Recurrent Search Algorithm for Depth Generalization

The results show that different thinking mechanisms can learn distinct computational solutions, even at similar ID performance, and in this setting, latent recurrence supports a reusable forward-search algorithm that generalizes beyond the training depth.

Hu-Zi Cheng, Zhe-Wei Zhang · 1 citation
#artificial intelligence Preprint Aug 2026

Understanding Curriculum Learning in Large Language Models via Cross-Difficulty Optimization Dynamics

It is shown that the transfer relationship between different difficulty levels characterizes the optimization dynamics induced by curriculum learning, which in turn explains the effectiveness of different curriculum schedules, and formalize this relationship as Relative Transfer, a principled measure of cross-difficult...

Zhikai Ding, Zi-Yi Ye · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.