Audiovisual Material Perception in Virtual Reality

Abstract

Material perception is a multisensory process, but how users resolve conflicts between modalities remains unclear. Here we examined material categorization in virtual reality when impact sounds and visual textures conflicted. Participants categorized the material of target objects that varied in both visual appearance and impact sound across four materials: glass, metal, plastic, or wood. Experiment 1 studied the effects of audiovisual pairing and agency (observation vs. direct interaction). Participants (N=42) were more likely to classify materials based on auditory cues. Visual dominance was rare, but stable within an individual when present. Agency had no significant effect. However, four specific audiovisual pairings revealed potential audiovisual illusions. In Experiment 2 (N=54), we varied the material of the tool striking the test object, and level of additive white noise. Noise had little effect. The rod material significantly influenced material choice. Thus, the sound profile of the complete acoustic event, not just that of the target object, determined categorization. For VR simulations, audio noise and agency appear to have little effect on material identification. However, mismatched audiovisual materials can produce non-intuitive or surprising categorization that undermines material perception.