Structured Thoughts For Improved Reasoning And Context Pruning
TL;DR
This work introduces Structured Thoughts, a framework that organizes reasoning into alternating blocks and blocks that captures exploratory scratch work, while the distilled conclusion of that step contains the distilled conclusion of that step.
Abstract
Large language models (LLMs) excel at generating long chains of thought, but long reasoning traces are often verbose and memory-inefficient. In this work, we introduce Structured Thoughts, a framework that organizes reasoning into alternatingandblocks:captures exploratory scratch work, whilecontains the distilled conclusion of that step. We construct a dataset of structured thoughts by segmenting reasoning traces intoblocks and prompting an LLM to summarize each step into its corresponding. Fine-tuning pretrained foundation models on this reformatted data produces models that adopt the structured reasoning style, leading to performance gains of up to 8.08\% on reasoning benchmarks compared to standard SFT. The explicit structure also enables context pruning: after each/pair, thecan be pruned, allowing the model to retain conclusions without keeping the full scratch work in the context. A proof-of-concept pruning implementation achieves an average of 85\% memory / context savings with an 8.67\% performance drop across mathematical tasks.