Publication: Quantifying the speech-gesture relation with massive multimodal datasets: informativity in time expressions
Authors
Pagán Cánovas, Cristóbal ; Valenzuela Manzanares, Javier ; Olza, Inés ; Ramscar, Michael ; Alcaraz Carrión, Daniel
item.page.secondaryauthor
Facultades de la UMU::Facultad de Letras
item.page.director
Publisher
Public Library of Science
publication.page.editor
publication.page.department
DOI
https://doi.org/10.1371/journal.pone.0233892
item.page.type
info:eu-repo/semantics/article
Description
Abstract
The development of large-scale corpora has led to a quantum leap in our understanding of speech in recent years. By contrast, the analysis of massive datasets has so far had a limited impact on the study of gesture and other visual communicative behaviors. We utilized the UCLA-Red Hen Lab multi-billion-word repository of video recordings, all of them showing communicative behavior that was not elicited in a lab, to quantify speech-gesture co-occurrence frequency for a subset of linguistic expressions in American English. First, we objectively establish a systematic relationship in the high degree of co-occurrence between gesture and speech in our subset of expressions, which consists of temporal phrases. Second, we show that there is a systematic alignment between the informativity of co-speech gestures and that of the verbal expressions with which they co-occur. By exposing deep, systematic relations between the modalities of gesture and speech, our results pave the way for the data-driven integration of multimodal behavior into our understanding of human communication.
publication.page.subject
Citation
PLoS ONE, 2020, Vol. 15(6): e0233892
item.page.embargo
Collections
Ir a Estadísticas
Este ítem está sujeto a una licencia Creative Commons. http://creativecommons.org/licenses/by/4.0/