Self-supervised skeleton action recognition based on graph prototype learning

Dual-Prototype Disentanglement Learning for Vision-Language Skeleton Representation

Auxiliary visual and language modalities can improve self-supervised skeleton action representation learning by supplying object, scene, and semantic cues that joint coordinates lack. Existing cross-modal training signals are often defined at the sample level or aligned in a single global space, making them sensitive to noisy external features and prone to suppressing skeleton-specific cues. This paper proposes DVLS, a disentangled vision-language-guided skeleton representation framework built upon a prototype-augmented baseline. DVLS splits each projected modality feature into shared and private subspaces: shared dimensions use cross-modal prototypes to capture transferable semantics, whereas private dimensions use modality-wise prototypes to preserve modality-specific structure. This design reduces the adverse effect of noisy global alignment while retaining external vision-language supervision. Experiments on NTU RGB+D 60, NTU RGB+D 120, and PKU-MMD show consistent gains over a strong global-prototype baseline on four of five skeleton-only linear protocols, including +1.18 points on PKU-MMD XSub and +0.39 points on NTU120 XSub, while matching the baseline on NTU60 XView. Under 1% semi-supervised NTU60 XView, DVLS improves the baseline from 72.04% to 73.35%.

MEMC: Masked modeling with efficient and minimal contrastive learning for self-supervised skeleton-based action recognition.

This work proposes MEMC (Masked Modeling with Efficient and Minimal Contrastive Learning), a novel framework that adopts an efficient sequential cascade strategy based on layer-grafted pretraining and introduces two CL enhancements to improve the discriminative capability of the learned representations.

CrossVLS: Cross-Modal Vision-Language Prototypes for Self-Supervised Skeleton Action Representation Learning

Artificial intelligence (AI)-driven positioning and tracking systems combine geometric trajectories with behavior understanding in smart-city, healthcare, and autonomous environments. Skeleton sequences provide compact, privacy-preserving motion geometry, but labeled data are costly, and coordinate-only self-supervision cannot recover object, scene, or interaction cues. Vision-language transfer can supply these cues, but instance-level targets remain sensitive to noisy crops, incomplete descriptions, and ambiguous actions. We propose CrossVLS, which is a cross-modal vision-language-guided framework that transfers semantic knowledge from red, green, and blue (RGB) frames and generated language descriptions to a skeleton encoder during pretraining while retaining skeleton-only inference. CrossVLS replaces noisy instance-level transfer with a shared prototype space: skeleton, RGB, and language features are softly assigned to a common prototype bank through balanced optimal transport, and the resulting assignments define semantic soft targets for contrastive learning. A full-batch progressive training schedule gradually increases cross-modal guidance without splitting the physical batch, preserving the support set used to construct semantic targets. Experiments on NTU RGB+D 60, NTU RGB+D 120, and PKU-MMD demonstrate strong performance under linear and semi-supervised evaluation using only the pretrained skeleton encoder at inference. These results show that prototype-mediated vision-language transfer can improve skeleton representations for the behavior-interpretation stage of positioning and tracking pipelines.

Fine-Grained Action Recognition with Cross-Attentive Latent Sparse Experts

Fine-grained human action recognition (FHAR) must distinguish visually similar actions that differ mainly in body configuration, timing, or local appearance. RGB representations retain visual context but often suppress joint-level geometry, whereas skeleton representations encode kinematics but discard dense spatial detail. We introduce FineX, which factorizes fine-grained cues into RGB appearance, pose heatmap geometry, and skeletal-graph topology. Pairwise cross-attention enables symmetric, stream-preserving information exchange, followed by a streamwise latent sparse Mixture-of-Experts that routes each representation to a content-dependent subset of shared experts, regularized by a load-balancing objective. FineX achieves state-of-the-art results on Gym99, Gym288, and Diving48. On the long-tailed Gym288, it raises mean class accuracy from 68.6% to 76.2% (+7.6 points) without textual supervision or large-scale vision-language pre-training, demonstrating the benefit of structured visual-pose-graph fusion and conditional expert refinement for FHAR.

HPCLR: hierarchical part-aware contrastive learning for skeleton-based action recognition

Visual Anchoring in Diffusion: Multimodal Zero-Shot Skeleton Action Recognition

Zero-shot Skeleton Action Recognition (ZSAR) remains ambiguous when unseen actions share similar skeleton joint dynamics but differ in objects or scene context. RGB provides these missing cues, yet existing multimodal methods typically maintain independent skeleton and RGB scoring branches and fuse their outputs. Without using unlabeled test data for adaptation or fusion calibration, a fixed fusion weight cannot capture class-pair-dependent modality reliability, while an adaptive rule lacks target-side feedback for deciding which branch should dominate. We bypass this weight-selection problem via the classify-by-generation paradigm, where each class is scored by how accurately a text-conditioned denoiser predicts the noise added to the skeleton feature. This formulation separates the progressively corrupted skeleton from fixed conditioning, allowing RGB and text to jointly condition a single class-scoring function rather than produce independent scores. We instantiate this idea as Multimodal Triplet Diffusion for Skeleton-Text Matching (TDSM-MM), augmenting a text-conditioned denoising Transformer with a non-diffused RGB condition token that serves as a stable visual anchor during skeleton data reconstruction. Our proposed TDSM-MM has been ablated via extensive experiments and achieved the best inductive accuracy on three of four NTU-60/120 splits and surpasses the transductive state-of-the-art on NTU-120 96/24 (i.e., 71.3% vs. 69.1%), without test-time adaptation, suggesting that diffusion-based methods can be a promising direction for zero-shot learning.