Skip to content
Book Open access

Beyond Bigger Models: Building Reliable Surgical AI from Limited Data

Nov 2026 · Proceedings of the 3rd International Workshop on Multimedia Computing for Health and Medicine · 0 citations · 3 references

Abstract

Recent advances in deep learning and foundation models have created a strong improvement in the analysis of surgical images and videos. However, in clinical applications, surgical AI still faces fundamental limitations: expert annotations are scarce and expensive, clinically relevant classes and events are often highly imbalanced, data cannot be easily shared across institutions, and models trained in the general domain may fail when used with special medical data. In this keynote, we will discuss recent work that explores alternative strategies for building reliable surgical AI under these constraints. Starting from the importance of carefully curated and annotated surgical datasets, such as GynSurg [5], we will see how unlabeled video data can be exploited through reliable semi-supervised learning. In particular, dual-invariance self-training with temporal and transformation consistency is used to identify trustworthy pseudo-labels for surgical phase recognition [6]. We will discuss the role of foundation models in the segmentation of medical images. Rather than assuming that large pretrained models can be deployed directly, recent results show that domain-specific adaptation remains critical: zero-shot text-prompted SAM 3 performs poorly in specialized cataract surgery scenes (e.g., Cataract-1k [1]), while targeted fine-tuning leads to dramatic improvements [2]. Foundation models can also serve as powerful teachers within federated semi-supervised learning, allowing lightweight models to learn from private, institution-specific data without centralizing raw data [4]. Finally, we will discuss synthetic data generation as another means of overcoming limited and imbalanced training datasets. Our recent results indicate that domain-specific generative models and semantic enhancement can be more valuable than simply scaling up general-purpose generative models, particularly when rare but clinically meaningful configurations need to be represented [3]. The conclusions for medical multimedia AI are as follows: progress will depend not only on bigger models but on combining limited expert supervision, trustworthy self-learning, domain adaptation, privacy-preserving collaboration, and targeted synthetic data. The central challenge is not to build models that can do more, but to build systems whose predictions can be trusted in the clinical domain.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.