Auditory Feedback as a Communication Modality: Modeling Perceived Semantic Expression of UX Sounds
TL;DR
This work identifies recurring audio design patterns and investigates how they relate to perceived semantic expression across the communication levels of system status, affect, and brand identity, then applies explainable machine learning to a predictive model capturing interpretable parts of this relationship.
Abstract
While multimodal interaction research has achieved significant progress in modeling speech and vision, structured approaches to nonverbal auditory feedback remain underexplored. UX sounds are short nonverbal cues that are vital for communicating system states, affect, and brand identity. However, their design is still guided largely by practitioner experience rather than empirical analysis. As alternative, we present a data-driven, interpretable framework for modeling the perceived semantic expression of UX sounds from a sonic interaction design perspective. Drawing inspiration from Natural Language Processing, we represent UX sounds as sequences of discrete events across five design layers: texture, timbre, accentuation, rhythm, melody. Using unsupervised sequence modeling on a corpus of 805 real-world UX sounds from seven industry sectors, we identify recurring audio design patterns and investigate how they relate to perceived semantic expression across the communication levels of system status, affect, and brand identity. Explainable machine learning is then applied to a predictive model capturing interpretable parts of this relationship, linking communication dimensions back to audio design patterns and providing design insights for auditory feedback in multimodal human-centered systems.