Intelligent Prompt Construction for Large Language Models in Knowledge-based Visual Question Answering

Abstract

Large Language Models (LLMs) have demonstrated strong capabilities in knowledge-based Visual Question Answering (VQA). However, existing prompt construction methods are often rigid and fail to fully exploit the reasoning potential of LLMs. To address this limitation, we propose the Intelligent Prompt Construction Framework (IPCF), which equips an autonomous agent with the ability to dynamically generate task-specific prompts. IPCF consists of a planner and a toolbox: the planner, powered by an LLM, enables autonomous decision-making, while the toolbox provides three tools—the vanilla VQA model for inspiration, the LLM for knowledge injection, and a knowledge base for information retrieval. This architecture allows the agent to flexibly determine when and how to invoke each tool and to construct adaptive prompts accordingly. Experimental results show that IPCF achieves performance gains of 2.6 and 1.9 points over existing baselines on the OK-VQA and A-OKVQA datasets, respectively.