Multisensory cues for conversational coordination

点击翻译按钮获取中文摘要

The primary unit of conversation is the turn. When two or more people converse, one speaker holds the floor and then almost seamlessly another person's turn begins. The turn exchange times are usually very short; the modal time in numerous studies is less than 200 ms with tight variation around this value. Such timing distributions are consistent across a wide range of languages and contexts. This fact is remarkable when the diversity of linguistic structures for the more than 5000 living languages is considered. Turn taking has been investigated in non-human primates and birds and similarities have been noted. Further, turn taking has been identified as a critical aspect of the development of language and communication. These facts suggest that the coordination of turn exchanges is a fundamental aspect of language. In spite of the primacy of this behaviour in many communication systems, much is unknown about the behavioural and neural representations needed to accomplish it. The approach taken in the proposed research is to quantify conversational behaviour in naturalistic situations and use innovative methods to manipulate aspects of the exchanges. The proposed studies include monitoring auditory-only and audiovisual conversations. In addition, unique perceptual and eye tracking studies will address what information aids rapid turn exchanges. Finally, our studies will use unique methods to produce real time perturbations of conversational dynamics. These methods include mapping the delay function between talking and listening and using real-time animation to modify the information that talkers display in conversation. By emphasizing the rich dynamics of conversation, the research will focus on the temporal constraints that shape our most common use of language. How time pressure and cognitive resource sharing determine the form and capacity of language use can only be truly studied by research about conversation. The methods that are required for this research involve non-invasive motion and acoustic measures of individuals talking naturally. New uses of animation and eye trackers will enhance research in this area. Understanding conversation has both engineering and basic science importance. The use of avatars and conversational agents is expanding rapidly and the importance of realism in these agents is obvious. The studies make novel contributions to language processing methods and research questions about language representation. Further, the studies provide the tools and statistical basis for understanding a number of clinical disorders of communication.