EcoReason: A Graph-Guided Evolutionary and Negative-Aware Reinforcement Learning Framework for LLM Commonsense Reasoning

Abstract

While large language models (LLMs) have achieved impressive gains in commonsense reasoning, they often fall into “associative shortcuts,” failing to distinguish correct answers from plausible but constraint-violating hard negatives. This reliance on semantic priors rather than specific situational constraints limits their fine-grained reasoning capabilities. To address this issue, we propose EcoReason (Evolved Commonsense Reasoning), a graph-guided evolutionary and negative-aware reinforcement learning (RL) framework. First, we introduce Graph-Guided Data Evolution, an iterative data generation strategy coupled with the student model’s training progress. In each round, we use knowledge graphs (KGs) to identify deceptive sibling concepts and employ a teacher LLM to create constraint-heavy question answering (QA) data targeting the student’s current blind spots. As training progresses, the synthesized curriculum becomes increasingly challenging. Second, we propose Negative-Aware Policy Optimization (NAPO), an RL algorithm built upon Group Relative Policy Optimization (GRPO). NAPO identifies “stubborn negatives,” defined as incorrect options that are repeatedly selected by sampled policies within the same rollout group, and applies stronger penalties to these recurring distractor-specific errors. Experiments show that EcoReason substantially improves LLM commonsense reasoning in complex constraint-heavy scenarios.