CIDA: Class-wise Reliability Gating for Dual-Stream Few-Shot Adaptation of Vision-Language Models
Abstract
Parameter-efficient adaptation of vision-language models such as CLIP typically modifies one side of the model, either the text prompt or the image features. When both sides are adapted, their predictions are merged with a weight fixed for the whole task, which cannot express that either branch may be more trustworthy for a particular query or class. We present CIDA (Class-wise reliability-gated Input-Dual-stream Adaptation), which adapts the visual input and the text representation in parallel over a frozen CLIP backbone and fuses the two streams with a gate driven by two complementary reliability signals: per-sample prediction statistics (entropy, classification margin, cross-stream agreement) and per-class support-set statistics (visual prototype compactness, image-text alignment). A similarity against the frozen class text embeddings is retained as a partially frozen text anchor against few-shot drift. On the standard 11-dataset suite with a frozen CLIP-RN50, CIDA averages 65.00% top-1 at 1 shot and 74.24% at 16 shots. Because baselines were obtained under other splits and training protocols, the comparisons are indicative rather than controlled. Three-seed repetition on EuroSAT, DTD and OxfordPets gives 85.49 ± 0.65%, 67.06 ± 1.15% and 85.17 ± 2.00%. Seed-paired ablations on EuroSAT isolate the gate: forcing a fixed 0.5/0.5 split costs 0.58 points (95% interval [0.18, 0.99]), whereas removing the learned text offsets costs 0.07 points ([−0.27, 0.13]). The gate is worth its place, but not because it admits text evidence: over ten datasets at 16 shots with three seeds each, the gate is strongly visual-dominant on every run, with four datasets collapsing to text weights < 10−5 and nine below 0.001, unchanged when the text branch is frozen CLIP. We attribute this to a scale asymmetry between an unbounded visual residual and a bounded text similarity, making branch calibration the next step rather than a larger text module.