Skip to content

Mastering Atari 2600 Games with Discovered Options

Oct 2026 · 0 citations · 94 references
Computer Science

TL;DR

Wayfarer is presented, a general, domain-agnostic, online deep RL agent that discovers options through Laplacian representation learning from high-dimensional observations and leverages them for control, enabling substantially faster learning of complex policies.

Abstract

Temporal abstractions, often instantiated as options, have long been regarded as a mechanism for accelerating credit assignment, facilitating exploration, and enabling generalisation in reinforcement learning (RL). However, developing general option discovery methods that are effective in large-scale, high-dimensional domains remains a fundamental challenge. Existing option discovery methods are either confined to relatively simple domains, depend on handcrafted or quasi-symbolic representations, or offer little improvement over learning without options. We present Wayfarer, a general, domain-agnostic, online deep RL agent that discovers options through Laplacian representation learning from high-dimensional observations and leverages them for control. We show that the resulting options simultaneously improve exploration, accelerate credit assignment, and generalise effectively to unseen settings, enabling substantially faster learning of complex policies. Wayfarer achieves state-of-the-art performance among single-stream agents on the most challenging Atari 2600 games, with the largest gains in games that require long-horizon exploration and strategic behaviour, such as Montezuma's Revenge and Private Eye.

View source

Similar papers

#artificial intelligence Preprint Oct 2026

Decoupling Exploration from Optimization in RLVR

Modern language models undergo reinforcement learning with verifiable rewards (RLVR) on top of already-trained checkpoints. A key promise of RLVR is the discovery of new reasoning strategies. In principle, a model can sample novel ideas absent from its prior training data. In practice, however, augmenting RLVR with str...

Saif Punjwani, Micah Goldblum · 0 citations
#artificial intelligence Open access Oct 2026

AdaC-GRPO: Towards Mastering Long-Horizon Visual Sudoku Puzzles via Adaptive Curriculum Learning

While reinforcement learning with verifiable reward (RLVR) enhances reasoning in large language models, its efficacy in long-horizon multimodal tasks remains underexplored. Visual Sudoku puzzles pose a representative challenge, requiring multi-step deductive reasoning over both local constraints and global dependency c...

Si-Da Li, Yuan-Xin Liu, Lin-Li Yao et al. · 0 citations
#artificial intelligence Preprint Oct 2026

ThunderSyncRL: Lossless Acceleration of Agentic Reinforcement Learning

This work introduces ThunderSyncRL, which starts gradient computation as soon as all required inputs are fixed, without policy staleness, and proves that gradient streaming produces the same GRPO and OPD updates as batch-synchronous training, without changing either objective.

Seil Kang, Hangoo Kang, Tarun Suresh et al. · 0 citations
#artificial intelligence Preprint Oct 2026

TrajEvolve: Trajectory Evolution for Reinforcement Learning with Hindsight Credit Assignment

TrajEvolve reformulates step-level continuous credit assignment as a binary classification task, whether each step is necessary for task completion, and simplification not only reduces the learning difficulty but also enables step labels to be obtained through counterfactual verification, achieving precise identificati...

Yi Wen, Hao Chen, Wan-Yu Wang et al. · 0 citations

Going Beyond State-Reaching: Learning Abstractions for Intrinsically Motivated Option Discovery

This work introduces an algorithm that identifies a small, relevant subset of features for each subgoal, yielding options that generalize broadly and accelerate exploration in three sparse-reward, image-based domains, including the Atari game MontezumasRevenge.

Akhil Bagaria, Anita de Mello Koch, G. Konidaris · 1 citation

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.