CoPL: Counterfactual Offline Policy Learning via Large Language Models

Abstract

Offline reinforcement learning is limited by the coverage of the offline dataset, which makes it difficult for policies to generalize to unseen goals and behaviors. We introduce CoPL, a framework for counterfactual offline policy learning via large language models. Unlike prior relabeling approaches that modify only instructions or action annotations, CoPL generates full counterfactual trajectories from real trajectory seeds, including states, actions, and their temporal evolution. Specifically, CoPL preserves the initial state of a real trajectory, rewrites the original goal into a semantically valid but behaviorally distinct counterfactual goal, and synthesizes the corresponding goal-conditioned rollout. The generated trajectories are then repaired and filtered before being merged with real offline data for policy learning. This design enables offline RL to learn from behavior patterns that are absent or weakly represented in the original dataset. Experimental results on robotic manipulation tasks show that CoPL substantially improves generalization on unseen tasks under both semantic and linguistic shifts, including a 55.8% relative improvement on the CLEVR-Robot CF-Easy setting, from 17.2% to 26.8% success rate.