This work introduces PULSE (Paired Utility Localization over Sparse Encodings), an SAE-based framework for identifying model-internal features associated with demonstration utility and using them for demonstration selection, and PULSE-Retriever, which converts its magnitude into a feature-relevance mask for scalable pool-scale retrieval.
Abstract
In-context learning is highly sensitive to demonstration choice, yet most methods select demonstrations using external query-demonstration similarity. Such criteria can miss model-specific signals: Similar demonstrations may activate different internal features and downstream behaviors. We introduce PULSE (Paired Utility Localization over Sparse Encodings), an SAE-based framework for identifying model-internal features associated with demonstration utility and using them for demonstration selection. Using a small labeled discovery set, PULSE samples candidate demonstration sets, measures their zero-shot-relative utility under the target model, and scores SAE features by how their activation differences align with utility differences. The top positive and negative coordinates form a sparse utility-localization vector. We use this vector in two complementary ways: as a signed score for controlled complete-set ranking, and as PULSE-Retriever, which converts its magnitude into a feature-relevance mask for scalable pool-scale retrieval. Across classification, generation, and reasoning benchmarks, PULSE-Retriever improves over the strongest baseline by 2-3 accuracy points, 0.6-0.9 BLEU-4, and 3.2 exact-match points, respectively, while controlled ranking validates the identified features encode a predictive set-level utility signal. Feature inspection and cross-dataset experiments suggest that the identified features capture task-relevant, dataset-conditioned patterns, yet retain utility signals that partially transfer across datasets. Our code is available at https://github.com/aohenuo/PULSE.
A transition atlas of triples is constructed, where a residual-state feature and an MLP-update feature jointly predict a target residual feature, and validate candidate triples by ablating the decoded update feature.
Hendrik Droste, C. M. Adriano, Kathrin Korte et al.· 0 citations
Performance-Driven Demonstration Selection (PDDS), which directly aligns demonstration selection with ICL performance, is proposed, which formulates selection as predicting the target LLM’s downstream task performance for a given query–in-context pair, replacing proxy heuristics with a performance-aware objec-tive.
Wen-Qiang Wang, Ming-Bo Yang, Ai-Ping Zhang et al.· Proceedings of the Thirty-Fi...· 0 citations
Overall, the results show that targeted feature quality depends jointly on the model signal and estimator, with detection and intervention capturing complementary properties.
Jonathan Drechsel, Steffen Herbold· arXiv.org· 0 citations
In-context learning (ICL) is crucial for boosting the inference performance of large language models (LLMs). However, the effectiveness of ICL in LLMs is greatly influenced by the choice of demonstration sets. Exhaustive searches over these sets are combinatorial, and existing selectors often rely on relevance or likel...
Sparse autoencoders are widely used to uncover interpretable features in neural networks, yet reliable recovery remains difficult when features overlap or activate infrequently. These challenges involve both inferring which features explain an input and learning the dictionary that represents them. Here, we unify infer...
Hadi Vafaii, Tejas Rao, David Chanin et al.· 0 citations
ClasSAE, a novel method that both automatically assigns classes to features and guides the encoder toward class-separable representations during training, is introduced, and three variants for enforcing sparsity within this framework achieve comparable overall performance with slightly different trade-offs.
Jakub Stępień, M. Mazur, J. Tabor et al.· 0 citations
Machine-learning networks can learn without detailed instructions from a designer. Those that do may offer insight into how the brain reorganizes itself during learning.
MIT News · Artificial Intelligence· news.mit.eduSep 29, 2026
Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.