This work systematically investigate whether jointly optimizing the self-supervised and supervised objectives during training provides a better alternative, and finds that JT consistently improves data and training efficiency while being robust in low-label settings, while PFT is more reliable in more specialized domains.
Abstract
Self-supervision is a powerful technique for learning visual representations from unlabeled data. Existing techniques primarily adopt a two-stage approach for self-supervised learning (SSL): a pretraining stage on unlabeled data followed by a finetuning stage on labeled data. While this pipeline has demonstrated extreme effectiveness, the interaction between self-supervised and supervised learning objectives remains insufficiently understood. In this work, we systematically investigate whether jointly optimizing the self-supervised and supervised objectives during training provides a better alternative. We compare two training paradigms: (1) the aforementioned pretraining followed by finetuning (PFT) and (2) joint training (JT), where self-supervised and supervised losses are optimized simultaneously in the same network. Across eight representative SSL methods and diverse computer vision tasks on natural, medical, crisis response, and remote sensing data, we evaluate performance under varying percentages of labeled data. Our results reveal that the relative effectiveness of PFT and JT depends strongly on the task at hand, the availability of labeled data, and the complexity of the domain. We find that JT consistently improves data and training efficiency while being robust in low-label settings, while PFT is more reliable in more specialized domains. We further analyze representation quality, robustness, and cross-domain generalization, providing new insights into how self-supervised and supervised objectives interact during optimization. We establish a comprehensive empirical benchmark for hybrid SSL-based semi-supervised learning and offer practical guidance for selecting appropriate training strategies across diverse vision applications.
The availability of large-scale labeled datasets has driven advances in AI-based computer vision, yet supervised learning remains costly and impractical in domains where annotation is scarce. Self-supervised learning (SSL) addresses this by harnessing unlabeled data to learn rich, transferable representations without explicit supervision. This survey presents a seven-category taxonomy of self-supervised learning methods covering: 1) input reconstruction or restoration, 2) context prediction, 3) contrastive learning, 4) feature clustering, 5) self-distillation-based feature reconstruction, 6) redundancy reduction, and 7) masked image modeling, with coverage extended to recent methods that include DINOv2, I-JEPA, SparK, data2vec 2.0, V-JEPA, DINOv3, V-JEPA 2, V-JEPA 2.1, C-JEPA, PhiNet v2. We situate this work within the existing survey landscape by explicitly comparing our contributions with prior SSL reviews. Beyond method descriptions, we provide: a chronological timeline of SSL evolution from 2008 to 2026; a cross-paradigm comparative analysis evaluating all seven families along collapse risk, scalability, computational cost, and downstream transferability; a dedicated comparative analysis of anti-collapse mechanisms; critical limitations and trade-off analyses per method family; and systematic benchmarking evidence on ImageNet-1K, PASCAL VOC, COCO, and five public medical imaging datasets. We also contribute a practical method selection decision matrix, extended challenge discussions, and actionable open problems for future research.
Kriti Ohri, D. Sukheja, Purushottam Kumar et al.· IEEE Access· 0 citations
Data augmentation plays a central role in self-supervised learning, as the quality and diversity of augmented views strongly influence the learned representations. However, most existing self-supervised methods rely on fixed stochastic augmentation pipelines, while more adaptive alternatives often require expensive policy search, adversarial training, or additional optimization procedures. In this paper, we propose a lightweight learnable augmentation framework based on Extreme Learning Machines (ELM) for self-supervised visual representation learning. The proposed module predicts image-dependent transformation parameters and applies them through a differentiable augmentation operator, enabling joint optimization with the representation model while introducing minimal additional computational overhead. The framework is integrated into three representative self-supervised learning methods: SimCLR, BYOL, and SimSiam. Extensive experiments on CIFAR-10, CIFAR-100, and Tiny ImageNet show that the proposed method consistently improves linear evaluation performance relative to reproduced baselines across most settings. In particular, the method yields notable gains on CIFAR datasets and remains effective on the more challenging Tiny ImageNet benchmark. A per-class difficulty analysis further shows that the proposed augmentation strategy substantially improves performance on hard classes, indicating stronger robustness to challenging categories while maintaining competitive overall performance. In general, the results demonstrate that lightweight learnable augmentation can effectively enhance self-supervised representation learning across different frameworks and datasets.
We argue that learning visual representations without labels requires a training signal jointly complete across three non-overlapping objectives: semantic invariance across augmented views, patch-level spatial prediction, and representational non-degeneracy. We formalize these as the observation, prediction, and regularization principles and prove (i) that combining observation and prediction without regularization admits the constant encoder as a global minimizer under negative-free alignment; (ii) that the two objectives are gradient-complementary and structurally non-conflicting at the encoder output; and (iii) that the momentum encoder converges to the same fixed point as the online encoder and provides no collapse guarantee at convergence. Contrastive alignment provides only self-limiting collapse resistance, formalized via an explicit gradient-decay argument. Dropping prediction withholds the spatial training signal by construction; dropping observation forfeits cross-view semantic invariance by construction; at the scale we study, no pair substitutes for the third. Every major self-supervised method is a special case of a single unified energy decomposition. We pair every theoretical claim with a controlled experiment, including a patch-retrieval evaluation for the spatial consequence of prediction.
Experimental findings show that pretrained models outperform those trained from scratch in terms of accuracy, convergence speed, and robustness, and a unified framework is proposed to integrate both processes in a deep learning pipeline.
Kwame Nkosi· International Journal of App...· 0 citations
Visual foundation models are a cornerstone of image and video understanding but typically require large amounts of data and computation. The current scale required for pretraining visual foundation models may be unsustainable or unnecessary, and significant benefits arise when effective models can be obtained with fewer resources. To better understand how self-supervised learning (SSL) objectives behave under resource constraints, we conduct a controlled study of image and video SSL objectives under matched data, architecture, and compute budgets. We compare contrastive, reconstruction, feature-prediction, and diffusion objectives and evaluate both standalone and jointly trained image-video SSL formulations across a diverse set of image and video understanding tasks. Our results show that DINOv2-style pretraining consistently provides the strongest overall performance under limited resources. Furthermore, combining DINOv2 with video SSL objectives such as VideoMAE substantially improves image classification and segmentation performance, but degrades video tracking and camera-pose estimation performance, revealing an important tradeoff between semantic and geometric representation learning. These findings suggest that combining image and video SSL objectives can be beneficial in resource-limited settings, while highlighting the need for improved methods that better balance semantic, temporal, and geometric supervision.
Experimental results demonstrate that SSL outperforms traditional supervised learning in accuracy, scalability, and transferability, while also reducing data labeling costs.
Shalini Gupta· International Journal of Inn...· 0 citations