2023· International Journal of Innovative Research in Humanities & Technology· 0 citations
TL;DR
Experimental results demonstrate that SSL outperforms traditional supervised learning in accuracy, scalability, and transferability, while also reducing data labeling costs.
Abstract
Self-supervised learning (SSL) is transforming artificial intelligence by enabling models to learn from large amounts of unlabeled data. Instead of relying on manual annotations, SSL leverages inherent data patterns to generate pseudo-labels, making it highly scalable and efficient for modern AI systems. Techniques such as contrastive learning, masked modeling, generative pretraining, and clustering have shown strong performance across vision, language, and speech tasks. This study examines key SSL methods, architectures, and training strategies, while addressing challenges like computational cost, feature collapse, and data bias. It proposes a unified framework that combines contrastive and generative approaches for improved efficiency and representation learning. Experimental results demonstrate that SSL outperforms traditional supervised learning in accuracy, scalability, and transferability, while also reducing data labeling costs. Future directions include integrating multimodal, reinforcement, and continual learning to further enhance SSL systems.
Experimental findings show that pretrained models outperform those trained from scratch in terms of accuracy, convergence speed, and robustness, and a unified framework is proposed to integrate both processes in a deep learning pipeline.
Kwame Nkosi· International Journal of App...· 0 citations
This work systematically investigate whether jointly optimizing the self-supervised and supervised objectives during training provides a better alternative, and finds that JT consistently improves data and training efficiency while being robust in low-label settings, while PFT is more reliable in more specialized domains.
Nusrat Munia, Tyler Ward, Nishat Nayla et al.· 0 citations
Traditionally, continual learning has assumed access to labeled data, yet many real-world applications -- such as lifelong robotics -- require models to adapt continuously from unlabeled streams. This has led to the development of continual self-supervised learning (CSSL), a rapidly growing area that lacks a dedicated, systematic review. In this work, we present a comprehensive survey of CSSL for vision, with connections to emerging vision-language settings. First, we analyze existing evaluation protocols and highlight inconsistencies that hinder fair comparison. We then examine why self-supervised objectives exhibit improved robustness to catastrophic forgetting, relating this to task-agnostic representations and smoother loss landscapes. Next, we organize existing methods into a unified taxonomy based on their forgetting-mitigation strategies, including distillation, replay, regularization, architectural approaches, model merging, and objective-level adaptation. Finally, we identify open challenges such as scalability and the need for fast adaptability. We argue that advancing CSSL requires moving beyond small-scale benchmarks towards continual pre-training paradigms for large-scale systems.
Sergi Masip Cabeza, Alicja Dobrzeniecka, Jonathan Swinnen et al.· 0 citations
This work presents a generative continual learning framework based on growing self-organizing maps (GSOMs) that are augmented with learned distributional statistics as well as encoder-decoder models for class-incremental learning. The proposed approach enables exemplar-free replay using distributional statistical memory, which eliminates the need to store raw data. Each GSOM unit maintains its own mean, variance, and covariance estimates, which are subsequently used to generate synthetic samples for replay; in encoder-decoder configurations, these samples are then decoded back into the input space (via ancestral sampling) for subsequent training. Our method is fully unsupervised, as it does not rely on explicit task boundaries or class labels during training. Results across multiple benchmarks show that the proposed approach achieves performance competitive even with supervised state-of-the-art memory-based methods while consistently outperforming memory-free approaches. In several settings, our framework matches or exceeds existing baselines, particularly in challenging single-class incremental scenarios. We also provide baseline results for single-class incremental TinyImageNet and MiniImageNet, offering a useful reference for future work. This work highlights the effectiveness of an unsupervised, adaptive, topology-driven neural form of statistical replay as a scalable, flexible approach to continual learning.
Pujan Thapa, Alexander G. Ororbia, Travis J. Desell· 0 citations
The availability of large-scale labeled datasets has driven advances in AI-based computer vision, yet supervised learning remains costly and impractical in domains where annotation is scarce. Self-supervised learning (SSL) addresses this by harnessing unlabeled data to learn rich, transferable representations without explicit supervision. This survey presents a seven-category taxonomy of self-supervised learning methods covering: 1) input reconstruction or restoration, 2) context prediction, 3) contrastive learning, 4) feature clustering, 5) self-distillation-based feature reconstruction, 6) redundancy reduction, and 7) masked image modeling, with coverage extended to recent methods that include DINOv2, I-JEPA, SparK, data2vec 2.0, V-JEPA, DINOv3, V-JEPA 2, V-JEPA 2.1, C-JEPA, PhiNet v2. We situate this work within the existing survey landscape by explicitly comparing our contributions with prior SSL reviews. Beyond method descriptions, we provide: a chronological timeline of SSL evolution from 2008 to 2026; a cross-paradigm comparative analysis evaluating all seven families along collapse risk, scalability, computational cost, and downstream transferability; a dedicated comparative analysis of anti-collapse mechanisms; critical limitations and trade-off analyses per method family; and systematic benchmarking evidence on ImageNet-1K, PASCAL VOC, COCO, and five public medical imaging datasets. We also contribute a practical method selection decision matrix, extended challenge discussions, and actionable open problems for future research.
Kriti Ohri, D. Sukheja, Purushottam Kumar et al.· IEEE Access· 0 citations