Impact of visual mouth image presentation on speech perception in noise in normal hearing subjects and cochlear implant users

Abstract

This study investigated the impact of visual speech cues on speech perception in noise and subjective listening effort in cochlear implant (CI) users compared with normal-hearing (NH) individuals. Furthermore, the study compared visual information presented either by a computer-generated avatar or by video recordings of a real speaker. Thirteen NH participants and twelve bilateral CI users completed the Oldenburg Sentence Test (OLSA) in six conditions: auditory‑only, visual‑only, and audiovisual presentations using either video recordings (female speaker) or a computer‑animated avatar (male speaker). Speech reception thresholds (SRTs) in noise and subjective listening effort (LE) were measured. Visual‑only lipreading performance and self‑assessed lipreading skills were also recorded. Video‑based audiovisual presentation significantly improved speech perception compared with audio‑only presentation (NH: 1.6 dB; CI: 3.5 dB). CI users demonstrated substantially higher lipreading scores than NH participants (51% vs. 28%), and lipreading performance strongly correlated with self‑reported ability. Despite SRT improvements, audiovisual presentation yielded only marginal reductions in listening effort for CI users and none for NH participants. In contrast, the computer‑animated avatar provided no measurable audiovisual benefit for speech perception or listening effort in either group. Real‑speaker video recordings considerably enhance audiovisual speech perception in both NH individuals and CI users, particularly benefiting CI listeners. However, reductions in listening effort were minimal, underscoring the dissociation between speech perception and perceived listening effort. Incorporating audiovisual measures may improve ecological validity in clinical assessment of CI outcomes.