Ask Me Twice: A Longitudinal Dataset for Studying Change in AI Chatbot Responses

Abstract

AI chatbots change continuously, but snapshot evaluations cannot show when answers changed or whether models became more cautious, verbose, current, or restrictive. We present Ask Me Twice (AMT), a public, growing archive created by the Internet Archive’s Wayback Machine team. AMT submits curated questions daily to an evolving model roster and preserves outcomes with question, model, date, and operational metadata. As of September 17, 2026, the public API reports more than 267,000 responses to 130 questions from 42 models, covering July 18–September 17 and growing by about 5,000 responses per day. Access includes an Explorer, a read-only REST API, daily bulk exports, an OpenAPI schema, and a Model Context Protocol server. This paper describes the resource’s design, schema, access, research uses, and limitations. Unlike static benchmarks and naturally occurring chat logs, AMT supports repeated-measures comparisons of the same prompts across models and days.