Skip to content

PULSE: Identifying Demonstration-Utility Features with Sparse Autoencoders

Sep 2026 · 0 citations
Computer Science

TL;DR

This work introduces PULSE (Paired Utility Localization over Sparse Encodings), an SAE-based framework for identifying model-internal features associated with demonstration utility and using them for demonstration selection, and PULSE-Retriever, which converts its magnitude into a feature-relevance mask for scalable pool-scale retrieval.

Abstract

In-context learning is highly sensitive to demonstration choice, yet most methods select demonstrations using external query-demonstration similarity. Such criteria can miss model-specific signals: Similar demonstrations may activate different internal features and downstream behaviors. We introduce PULSE (Paired Utility Localization over Sparse Encodings), an SAE-based framework for identifying model-internal features associated with demonstration utility and using them for demonstration selection. Using a small labeled discovery set, PULSE samples candidate demonstration sets, measures their zero-shot-relative utility under the target model, and scores SAE features by how their activation differences align with utility differences. The top positive and negative coordinates form a sparse utility-localization vector. We use this vector in two complementary ways: as a signed score for controlled complete-set ranking, and as PULSE-Retriever, which converts its magnitude into a feature-relevance mask for scalable pool-scale retrieval. Across classification, generation, and reasoning benchmarks, PULSE-Retriever improves over the strongest baseline by 2-3 accuracy points, 0.6-0.9 BLEU-4, and 3.2 exact-match points, respectively, while controlled ranking validates the identified features encode a predictive set-level utility signal. Feature inspection and cross-dataset experiments suggest that the identified features capture task-relevant, dataset-conditioned patterns, yet retain utility signals that partially transfer across datasets. Our code is available at https://github.com/aohenuo/PULSE.

View source

Similar papers

#machine learning Preprint Sep 2026

Where Decoder Cosine Similarity Fails for SAE Feature Flow Discovery

A transition atlas of triples is constructed, where a residual-state feature and an MLP-update feature jointly predict a target residual feature, and validate candidate triples by ablating the decoded update feature.

Hendrik Droste, C. M. Adriano, Kathrin Korte et al. · 0 citations
Conference Open access Sep 2026

Performance-Driven Demonstration Selection for In-Context Learning

Performance-Driven Demonstration Selection (PDDS), which directly aligns demonstration selection with ICL performance, is proposed, which formulates selection as predicting the target LLM’s downstream task performance for a given query–in-context pair, replacing proxy heuristics with a performance-aware objec-tive.

Wen-Qiang Wang, Ming-Bo Yang, Ai-Ping Zhang et al. · 0 citations

You Only Edit Once: Incentivizing In-Context Capability of LLMs via Local Demonstration Refinement

In-context learning (ICL) is crucial for boosting the inference performance of large language models (LLMs). However, the effectiveness of ICL in LLMs is greatly influenced by the choice of demonstration sets. Exhaustive searches over these sets are combinatorial, and existing selectors often rely on relevance or likel...

Jia-Rong Wen, Qi Wang, Yun Qu et al. · 1 citation
#artificial intelligence Preprint Oct 2026

Inference and learning in sparse autoencoders as natural gradient flow

Sparse autoencoders are widely used to uncover interpretable features in neural networks, yet reliable recovery remains difficult when features overlap or activate infrequently. These challenges involve both inferring which features explain an input and learning the dictionary that represents them. Here, we unify infer...

Hadi Vafaii, Tejas Rao, David Chanin et al. · 0 citations
Preprint Oct 2026

ClasSAE: Class-Aligned Sparse Autoencoders via Differentiable Feature-Class Affinity

ClasSAE, a novel method that both automatically assigns classes to features and guides the encoder toward class-separable representations during training, is introduced, and three variants for enforcing sparsity within this framework achieve comparable overall performance with slightly different trade-offs.

Jakub Stępień, M. Mazur, J. Tabor et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Sep 29, 2026

Who we become when we talk to machines

Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.