Aug 2026· Frontiers of Computer Science· Vol 8· 0 citations· 66 references
TL;DR
It is demonstrated that LLMs speed agent learning and greatly reduce the human effort required to achieve robust, reliable, and repeatable task performance.
Abstract
Interactive Task Learning (ITL) enables cognitive agents to learn novel tasks (in one shot) from natural-language instruction and allows humans to customize agents to align with individual preferences. ITL relies on reasoning over and learning from multiple sources of knowledge, a strength of cognitive architectures. However, ITL requires frequent human input, which can be tedious and time-consuming. We evaluate large language models (LLMs) as an additional source of knowledge for ITL. We summarize initial experiments exploring the potential use of LLMs in ITL and then describe a novel method (STARS) that markedly improves the reliability of task learning from LLMs for embodied ITL agents. We demonstrate that LLMs speed agent learning and greatly reduce the human effort required to achieve robust, reliable, and repeatable task performance.
This research benchmarks human evaluations against a large language model (LLM) using a multi-agent approach and/or retrieval-augmented generation (RAG) to automate complex content analysis tasks to leverage artificial intelligence’s efficiency and precision alongside humans’ contextual understanding and domain expertise.
Xinyu Fu, Chaosu Li· Journal of Planning Educatio...· 1 citation
A novel RAIE taxonomy along four scaling dimensions is proposed, which optimizes the entire thought process through search algorithms and self-verification, and introduces a task-oriented guideline for choosing the best TTS strategy.
Jiayu An, Zheng Chen, Yongcheng Jing et al.· 0 citations
River, a simple training recipe that improves reward quality by filtering low-quality environments and augmenting outcome rewards with process-level behavior regularization is proposed, which achieves the best performance among evaluated open-source RL-trained 8B models across four terminal-agent benchmarks.
Yi-Fan Yao, Bo Pang, Xuan-Phi Nguyen et al.· 0 citations
Large Language Models (LLMs) have demonstrated strong capabilities in knowledge-based Visual Question Answering (VQA). However, existing prompt construction methods are often rigid and fail to fully exploit the reasoning potential of LLMs. To address this limitation, we propose the Intelligent Prompt Construction Framework (IPCF), which equips an autonomous agent with the ability to dynamically generate task-specific prompts. IPCF consists of a planner and a toolbox: the planner, powered by an LLM, enables autonomous decision-making, while the toolbox provides three tools—the vanilla VQA model for inspiration, the LLM for knowledge injection, and a knowledge base for information retrieval. This architecture allows the agent to flexibly determine when and how to invoke each tool and to construct adaptive prompts accordingly. Experimental results show that IPCF achieves performance gains of 2.6 and 1.9 points over existing baselines on the OK-VQA and A-OKVQA datasets, respectively.
Zhongjian Hu, Peng Yang, Dongmei Yang et al.· ACM Transactions on Multimed...· 0 citations
Large language model (LLM) agents have demonstrated remarkable proficiency in manually constructed environments, yet their performance frequently collapses when transitioned to complex real-world settings. Existing research largely attribute this degradation to the compositional generalization gaps in LLMs on combinations of multiple simple, well-structured environments. In this work, we propose that LLM web agents can learn simple environment observations at test time. Specifically, we introduce trial steps for agents to decompose a complex environment observation into sub-modules, and implement a label-free learning method, Test-Time Environment Decomposition (TTED), to adapt agent behaviors with experience during inference. Our empirical evaluations demonstrate the framework's efficacy across both synthetic and realistic benchmarks, showing (1) experience gains acquired within simpler sub-environments can be effectively composed to improve performance in the full one, and (2) test-time training on sub-environments can significantly enhance the compositional generalization of agents in real-world web automation tasks. We also provide key insights in the design of the label-free learning algorithm. As more complex environments are accessed by LLM agents, we believe learning environment decomposition skills at test time will be critical for robust real-world deployment.
Jun-Xuan Li, Zijun Liu, Zi-Yi Huang et al.· 0 citations
Agentic systems driven by large language models (LLMs) regularly feature two key mechanisms to autonomously solve complex problems: synthesizing text-based knowledge and procedures from past experiences and building parametric (weight-space) skill libraries for recurring sub-goals. To date, research has largely treated these as orthogonal pursuits: either organizing textual knowledge through composition and reflection, or consolidating parametric skills via weight-space merging. Consequently, the seamless integration of text and model weights for targeted performance improvements remains largely unexplored. This work bridges this modality gap by treating model weights as an additional modality that an LLM can natively reason over. We instantiate parametric learning via prefix-tuning and augment an LLM to ingest both prefix weights and rich textual data which capture relationships to a target capability. Our augmented LLM, which we call SkillSmith, synthesizes these inputs to perform instruction-steered parametric synthesis, directly outputting new prefix weights that manifest the target skill. We demonstrate that our approach significantly outperforms both text-only and weight-space-only baselines, unlocking performance gains that are out of reach for uni-modal (text-only or weight-only) adaptations.
Lucio M. Dery, Benedict Aaron Tjandra, Siavash Samiei et al.· 0 citations