A two-stage deep-learning framework for image classification of 46 southeast Asian fruit and vegetable categories

Abstract

Introduction The ThaiSook mHealth application was designed to promote a healthy lifestyle and mitigate risk factors associated with noncommunicable diseases; one of the administrative tasks involved is the manual flagging and reporting of incorrect entries related to fruit and vegetable intake. However, the process can be time-consuming and prone to errors. Automating food categorisation can provide immediate insights into dietary choices and encourage healthier eating habits. Therefore, we aimed to develop a food image classification model that can be used to categorise different types of Southeast Asian fruits and vegetables, for which nutritional data are available from the Thailand Ministry of Public Health, using the ThaiSook mHealth application. Methods A two-stage model encompassing binary and multiclass classification stages was developed to classify 46 types of Southeast Asian fruit, vegetables, and vegetable-heavy dishes. We used a two-stage strategy featuring binary and multiclass models; EfficientNetV2S emerged as the optimal choice for the classification model. The binary classification phase eliminated non-fruit and non-vegetable images, whereas the subsequent multiclass classification identified specific fruit or vegetable types. The methodology incorporates semi-supervised learning using Noisy Student training, complemented by techniques such as pseudo-labelling, label smoothing, and temperature scaling. These strategies enhance model performance and robustness and effectively exploit unlabelled data. Results The dataset included the labelled group (5,760 fruit and vegetable images and 5,450 non-fruit or vegetable images) and unlabelled group (29,385 images; contributed by ThaiSook application users). Our system’s comprehensive performance showed that the end-to-end pipeline achieved accuracies of 92.43, 94.56, and 94.91% in the top-1, −3, and −5 experiments, respectively. Conclusion We developed a robust image classification model tailored to fruits and vegetables that can be practically applied for dietary assessment using the ThaiSook mHealth application. Our findings demonstrate the resilience and suitability of our model for resource-constrained mobile applications. Trial registration The clinical trial was registered according to the WHO International Clinical Trials Registry Platform (WHO-ICTRP) in the Thai Clinical Trials Registry (ID TCTR20220611001) on 11 June 2022.