While trained models saw a drop in performance when tested on a new dataset, those pretrained on large diverse datasets outperformed limited target‐specific training even when deployment conditions were not represented during training, confirming training diversity improves transferability.
Abstract
Passive acoustic monitoring (PAM) generates large datasets that are costly to analyze manually, creating the need for accurate automated detection methods. While deep learning (DL) has shown promise for PAM‐related tasks, the transferability of trained models to unseen datasets remains a critical challenge. To address this, we evaluated four methodological approaches for maximizing the transferability of a DL sperm whale (
Physeter macrocephalus
) click detector: (1)
cross‐environment evaluation
(training on five datasets, testing on an unseen sixth), (2)
limited target‐domain data training
(model retraining using only a small, site‐specific dataset), (3)
pretraining with random fine‐tuning
, and (4)
pretraining with active fine‐tuning
(selecting fine‐tuning samples based on model uncertainty). While trained models saw a drop in performance when tested on a new dataset, those pretrained on large diverse datasets outperformed limited target‐specific training even when deployment conditions were not represented during training, confirming training diversity improves transferability. Fine‐tuning with 500 target recordings effectively mitigated performance drops when deploying a trained model to new datasets. Active fine‐tuning consistently outperformed all other approaches, although the improvement over random fine‐tuning was marginal. For new studies, we recommend using models pretrained on diverse datasets and fine‐tuning them with minimal target data.
Experimental findings show that pretrained models outperform those trained from scratch in terms of accuracy, convergence speed, and robustness, and a unified framework is proposed to integrate both processes in a deep learning pipeline.
Kwame Nkosi· International Journal of App...· 0 citations
This study systematically compares seven pre-trained feature extractors across three architectural families, convolutional neural networks (CNNs), Vision Transformers (ViTs), and self-supervised models to provide practical guidance on model selection for downstream deep learning tasks.
Rafeek Sibrikhan, M. Mufassirin· Sri Lankan Journal of Techno...· 0 citations
Hyperparameter optimization (HPO) can materially affect the performance of deep learning (DL) image classifiers, but there is little empirical guidance on how to derive the validation signal that drives it, especially for the small sample sizes common in fields such as medical imaging. We compared three HPO protocols in terms of {\em absolute performance-estimation error} (AEE; the absolute difference between the winning configuration's validation AUROC and its test AUROC): fixed holdout (F), reshuffled holdout (R), and 5-fold cross-validation (C). The search space, sampler, training procedure, architecture, and test set were held identical across protocols. We evaluated the protocols on three public datasets spanning two regimes: binary medical imaging (RSNA pneumonia radiographs and binarized HAM10000 skin lesions) and 200-class natural imaging (Tiny ImageNet), across a range of development set sizes $n$ and two backbones (ResNet-18 on all datasets, Vision Transformer (ViT-S/16) on RSNA). On the medical datasets, every point estimate favored cross-validation over both holdout protocols, with reductions in AEE largest at small sample sizes and diminishing as $n$ increased. This pattern remained robust under conservative family-wise adjustment. On Tiny ImageNet, AEE was negligible under all three protocols. Test AUROC was generally similar among protocols. Fixed holdout had lower mean AEE than reshuffled holdout in 11 of 12 medical conditions, although this secondary finding was less uniformly supported. For small-sample medical image classification, we recommend cross-validation-based HPO when computational resources permit because it trades additional computation for a more reliable development-time estimate of subsequent test performance.
Medical Vision-Language Models (VLMs) exhibit strong zero-shot performance, yet their effectiveness still declines on out-of-distribution (OOD) data due to domain shifts and class bias inherited from large-scale pretraining. Existing few-shot adaptation methods typically introduce additional trainable components, which can be unstable in extremely low-data regimes (e.g., 1-shot), and lack robustness on different medical data. We present TCLA, a purely training-free few-shot adaptation method for Medical VLMs, which is fast and model-agnostic. TCLA corrects inference logits based on a small set of support samples, boosting pretrained VLMs performance by improving inter-class deconfusion and reducing domain shift. Extensive experiments on nine datasets across multiple medical imaging modalities including X-ray, Ultrasound, MRI, CT, Histopathology, demonstrate that TCLA consistently improves OOD performance of Medical VLMs and, in most of cases, outperforms existing training-based adaptation methods.
This tutorial covers recent developments in OOD detection from both theoretical and practical perspectives, including four major categories: (1) post-hoc methods, (2) training-based methods with auxiliary outliers, (3) training methods without auxiliary outliers, and (4) foundation model-based approaches, along with recent advances in each area.
Suranga Seneviratne, Dishanika Denipitiyage, Sanjay Chawla et al.· Proceedings of the 32nd ACM...· 0 citations