Pretraining how a street is appraised is weak evidence that a model represents the street as the brain does, and prediction accuracy and neural correspondence were not reliably related across models, and the best predictor was among the least aligned.
Abstract
Pretrained vision embeddings are widely used to model how people appraise urban scenes, and they are validated almost entirely by how well they predict human ratings. Accurate prediction shows that an embedding contains the information needed to recover the ratings. It does not show that the embedding arranges scenes as the human visual system does, which is assumed when distances or dimensions in the embedding are read as perceptual. Using openly released EEG recorded while 63 adults viewed and rated street scenes of Berlin, we compared the neural representational geometry of the scenes with the geometry of pretrained vision models that differ in training objective and size, of simple image descriptors and of the ratings themselves, each relative to a noise ceiling given by the agreement between participants. No feature space reached more than about half of the noise ceiling, which corresponds to about a fifth of the reliable variance in the neural geometry. A descriptor of oriented edge energy reached the level of most pretrained models while capturing a different part of the neural geometry, and deeper layers corresponded to later neural responses. The same embeddings predicted held-out ratings well, up to r = 0.87, yet prediction accuracy and neural correspondence were not reliably related across models, and the best predictor was among the least aligned. Predicting how a street is appraised is therefore weak evidence that a model represents the street as the brain does.
A multi-dimensional observer model is proposed that represents images as distributions in a latent perceptual space and that models human judgment as comparing noisy samples, which finds that the perceptual spaces needed to account for image quality judgment in humans are extremely low-dimensional compared to the image...
Large self-supervised vision models learn representations that support scene segmentation and the semantic decomposition of physical objects. We ask whether their representational geometry supports transfer to visual reasoning problems without any task-specific fine-tuning. We hypothesized that relational representatio...
This narrative review examines deep learning for static visual scene classification across indoor, outdoor, and aerial imagery and concludes that useful progress should be assessed through reproducible gains under matched protocols and credible transfer to new environments, rather than isolated accuracy values.
Sandeep Kumar Aazad, Kritika Chaudhary, Taniya Saini et al.· Journal of Computing and Dat...· 0 citations
Test whether instruction-following vision-language models (VLMs) can generate human-like ratings in a zero-shot manner: using the original human-rating instructions as a single prompt without any task-specific training, showing that VLMs can replicate human semantic judgments without any training.
Katarzyna Jurewicz, Yohaï-Eliel Berreby, B. S. Krishna· bioRxiv· 0 citations
Vision-language models (VLMs) are increasingly used to measure urban change from repeated street-level imagery, and the results are often mapped for individual sample points. Such maps assume that a perception score changes only if the street does. We test this assumption with repeated Google Street View captures of un...
Computer vision saliency models predict where people will look, one map per image, and a billion-dollar predicted-attention industry sells those maps in place of measuring real viewers. I test the leading models from the audience side, against 11.4 million webcam gaze points from 3,023 US adults recruited to national q...
Elena Sirotkina· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.