Development and Evaluation of an Offline Vision-Language Model System for Estimating Nutrients from Meal Photographs on Personal Devices: The PLANS (Personal-device Local Assessment of Nutrients System) Study
TL;DR
An offline VLM on a tablet reproduced server-side energy accuracy and is a candidate tool for dietary counselling where photographs cannot leave the institution; given the wide limits of agreement, its usefulness in practice remains to be examined.
Abstract
Background: Estimating nutrient intake from meal photographs with vision-language models (VLMs) is now practical, but patient photographs often cannot leave the institution. We developed PLANS (Personal-device Local Assessment of Nutrients System), in which a small open VLM runs entirely on a personal device. Methods: Open VLMs were compared on two image sets with reference values (NutriImage, 887 Japanese meals; SNAPMe, 1,134 US meals), and the selected model (Qwen3-VL-4B, 4-bit) was run on an iPad Air without network access. Energy (primary outcome), protein, fat, carbohydrate and salt were evaluated by median absolute percentage error (MedAPE) and Spearman rank correlation (rho). The effect of supplying the food weight, including errors in it, was examined. Results: Energy estimates on the iPad were identical to those of a desktop-computer run for 84-86% of images (other values, 75-87%). MedAPE on the iPad was 37.0% and 36.5% for energy (26-28% of images within +/-20%), 43-65% for the macronutrients and 68-101% for salt (rho 0.51-0.85). On the desktop computer, supplying the true weight reduced MedAPE of the 4B model on NutriImage from 37.0% to 32.5% for energy and from 100.0% to 66.8% for salt; energy MedAPE stayed below the base value for declared weights of 0.5-1.25 times the true weight or rounded to 50 g, but rose markedly when the container weight was included. Conclusions: An offline VLM on a tablet reproduced server-side energy accuracy and is a candidate tool for dietary counselling where photographs cannot leave the institution; given the wide limits of agreement, its usefulness in practice remains to be examined.
