Accuracy of free-tier and subscription chatgpt versions for carbohydrate counting in frequently consumed Brazilian foods: a factorial validation study

Abstract

To evaluate the accuracy of ChatGPT carbohydrate estimates for commonly consumed Brazilian foods with standardised composition, and to compare free-tier with subscription-tier versions and simplified with elaborated prompts. A 2 × 2 factorial design covered the 20 most frequently acquired foods in the Brazilian Household Budget Survey (POF 2017–2018). Each food was queried once under four conditions: model tier (GPT-5.1 Instant, free; GPT-5.1 Thinking, subscription) × prompt complexity (lay user; expert). Following the ISO 15197 architecture, an estimate was clinically accurate when its deviation from the Brazilian Diabetes Society (SBD) reference was within 1.0 g per 100 g or 15%, whichever was more permissive. Analyses accounted for the clustering of four observations within each food (Bland–Altman for clustered data, within-food paired tests, linear mixed models with a food random intercept). ChatGPT returned usable estimates for all 80 queries; overall clinical accuracy was 85.0% (68/80; 95% CI 75.3–92.0). No evidence of differences was observed between model tiers or prompt strategies (85.0% in all four conditions; exact McNemar, p = 1.000). Mean absolute error ranged from 1.14 to 1.43 g per 100 g. Agreement analysis showed a small, non-significant tendency towards underestimation (− 0.37 g per 100 g; 95% limits of agreement − 4.42 to + 3.69; p = 0.389). Under controlled conditions, 85.0% of estimates fell within this accuracy margin, and no evidence was found that model tier or prompt complexity influenced estimation error. These findings do not establish clinical safety, effectiveness, equivalence between tiers, or suitability for unsupervised patient use.