The results show that state-of-the-art LLMs can solve much of the standard introductory physics problems, but that their performance remains uneven and constrained by problem modality and problem difficulty.
Abstract
Reasoning or inference-scaling models are the new generation of Large Language Models (LLMs) capable of complex problem solving. To investigate their problem-solving capability in physics, we evaluated model o4-mini by OpenAI on solving traditional, end-of-chapter problems from Halliday and Resnick's"Fundamentals of Physics,"spanning core topics in the undergraduate physics curriculum. Performance was analyzed across modality and problem difficulty. The model solved the problems with overall accuracy of about 90%, but performance depended strongly on representation: accuracy was much higher on text-only problems (96%) than on problems requiring coordinated interpretation of text and images (79%). Accuracy also declined significantly as the problem difficulty increased from low to medium to high. These results show that state-of-the-art LLMs can solve much of the standard introductory physics problems, but that their performance remains uneven and constrained by problem modality and problem difficulty.
Understanding how (multimodal) large language models perform on physics problems requires benchmarks that reflect the difficulty and breadth of expert-level physical reasoning. Existing physics benchmarks remain limited in the following two important ways: (1) short of high-difficulty datasets, and (2) lack of comprehensive coverage of visual forms, knowledge points, and step-by-step solution processes. As a result, model performance on current datasets may not be fully representative of their ability to solve complex physics problems. To address these issues, we present PhysElite, a large-scale bilingual multimodal benchmark for Olympiad-level physics reasoning. PhysElite contains 11,586 Olympiad-tier problems. For each problem, we provide corresponding visual diagrams, step-by-step bilingual Chinese-English solution derivations, and the final answer. We benchmark 18 open-source and closed-source MLLMs, and find that even the strongest model reaches only 33.7% answer accuracy. We additionally conduct step-level process evaluation to diagnose where models fail in the reasoning chain. Our datasets are released at https://huggingface.co/datasets/physelite/PhysElite.
Ruoran Xu, Wending Gao, Liyunfeng Chen et al.· 0 citations
This paper examines the cognitive foundations of problem solving in introductory physics, drawing on Daniel Kahneman's dual-process theory and research on judgment and decision making. Students often struggle with problem analysis, mathematical reasoning, and cognitive biases. To address these challenges, the paper proposes a six-step problem-solving framework grounded in dual-process principles and integrates it with structured problem-solving journals. The framework and journals were implemented in university physics courses, where students were required to make their reasoning explicit in written homework. In a controlled classroom study, students who consistently applied the six-step structure in written journals scored 24% higher on average on the final exam than peers who completed only online homework. These findings illustrate how slowing down intuitive responses and examining assumptions allows students to engage more fully in analytical problem solving.
A. Godunov· American Journal of Physics· 0 citations
This study presents an AI-powered system for generating isomorphic physics problems using large language model (LLM)-based agent workflows. The system is designed around three practical goals: preserving the same conceptual and problem-solving structure as the original problems, varying construct-irrelevant features such as scenarios and numerical values, and producing questions that are directly usable without expert revision. The workflow combines prompt chaining, agent-based verification, and automated LaTeX compilation within a publicly accessible website hosted on a Raspberry Pi. To evaluate the system, we developed an eight-item rubric and tested the system using 13 multiple-choice questions from a calculus-based introductory Newtonian mechanics course. The evaluation results showed that 89% of the questions generated were rated as fully specified and directly usable. However, the system also showed limitations. The results suggest that LLM-based systems have significant potential for reliable instructional problem generation while also highlighting important challenges for future development.
This study examines Artificial Intelligence (AI)-generated physics solutions from two connected perspectives: how prompt design shapes these solutions and how students can be prepared to critique them. Using a rotational-mechanics problem, we adapted a problem-classification framework to examine prompt variations, evaluating OpenAI's o4-mini responses with the Minnesota Assessment of Problem Solving (MAPS) rubric. Well-specified prompts improved solution completeness; underspecified and multimodal prompts exposed weaknesses in physics reasoning and correctness. In the student-evaluation phase, 24 introductory physics lab groups evaluated an o4-mini solution to this problem after either independently solving a related problem or critiquing its AI-generated solution with MAPS-based reflection questions. Problem-solving-only groups exhibited uncritical or misconception-based critiques; MAPS-guided groups identified more expert-aligned issues, including skipped numerical procedures and undefined notation. Together, our findings contribute to physics education research by showing how AI-generated solutions can ground both model-reasoning benchmarks and improved student critique of that reasoning through MAPS-based reflection.
N. Borse, Amir Bralin, S. Savage et al.· 0 citations
A unified framework is introduced that distills intermediate representations that explicitly encode the physical modeling process and adopt a two-stage post-training strategy, where supervised fine-tuning establishes structured modeling, and reinforcement learning with rubric-based feedback improves the quality of the modeling process.
Ye Zhang, Xuehang Guo, Rui Pan et al.· 0 citations