Interactions of perspective-taking in gesture and speech

Abstract

This dissertation investigates the role of perspective-taking in multimodal communication, with a particular focus on the interaction between speech and gesture. It explores whether perspectives expressed across modalities tend to align or diverge, whether gestures can be interpreted as quoted content in indirect discourse, and how a gesture's form and prototypicality affect whether it receives a perspective-dependent interpretation or not. The empirical foundation consists of five experimental studies. Experiments 1 and 2 examined the at-issue status of character viewpoint gestures and observer viewpoint gestures using acceptability- and felicity-rating tasks. The results indicate that both gesture types generally contribute not-at-issue meaning, but that character viewpoint gestures can shift toward the at-issue dimension when they provide information directly relevant to the discourse. Experiment 3 targeted the hypothesis that the perspectives simultaneously expressed in gesture and speech are preferably aligned, thereby controlling for potential alignment-blocking factors identified by Hinterwimmer, Patil & Ebert (2021). The findings revealed no systematic tendency toward alignment, suggesting that the two modalities can maintain independent perspectives and thus allow for multiperspectivity. Experiments 4 and 5 investigated whether perspective-dependent elements such as self-pointing gestures and face emoji can receive shifted interpretations, that is, interpretations from the reported speaker, in indirect discourse. The results obtained from the two experiments confirm this. It is proposed that these findings are best analyzed as instances of mixed quotation. Quotation is analyzed as a form of demonstration in the sense of Clark & Gerrig (1990), that is, as an iconic mode of communication that depicts certain aspects of its referent—the original utterance conveyed by the indirect discourse. Finally, the dissertation extends the supplemental account of gesture semantics (Ebert & Ebert, 2014) to the verb phrase domain. Here, a distinction is drawn between prototypical gestures, which only carry those features that are characteristic of a prototypical version of the event they represent and thus are interpreted as abstract event types, and non-prototypical gestures, which modify this canonical form by adding further, contextually specific features, thereby receive interpretations as concrete event tokens. In the proposed analysis, non-prototypical gestures stand in a viewpoint-dependent similarity relation to the event they depict, whereas prototypical gestures do not. This extension captures how viewpoint shapes the interpretation of gestures in the verb phrase domain and parallels the analysis with analogous concept- and object-related interpretations observed in the noun phrase domain. Drawing on these findings, the dissertation marks a starting point for the development of a unified, perspective-sensitive formal theory of demonstrations that could account for multimodal quotation phenomena, the interpretation of iconic gestures as well as other iconic enrichments. More broadly, the results underscore the need for semantic theories to treat visual and verbal signals as equal contributors in explaining how meaning is constructed.