Can Frontier LLMs Capture $40 Billion in Supply Chain Value?
Abstract
Large Language Models (LLMs) perform well on many NLP tasks, but their ability to reason about complex supply chain relationships remains unclear. We introduce the Supply Chain Intuition Exam (SCIE), a benchmark for evaluating whether LLMs can identify and infer direct and indirect supplier relationships. We compare standard LLMs, models with grounding-based retrieval, and deep research (DR) systems. Despite access to public financial reports and real-world data, recall remains low: standard and grounded models achieve approximately 3--16%, while the best DR systems reach only 22%. These results suggest that retrieval alone is insufficient for professional-grade supply chain reasoning. SCIE highlights the need for methods that better integrate domain knowledge, retrieval, and multi-step reasoning in business and finance applications.