Concurrency, synchrony, and interaction in multimodal communication

人机交互在我们的生活中变得越来越普遍。最终，我们预测，这种互动将是语言上的交流，并将需要机器模拟人类的熟人。简单的界面，如电话上的语音识别，将被与以多种信号/感官模式操作的物理代理的交互所取代。一个明显的问题是，如果人类和机器各自的行为领域达到相当程度的匹配，它们之间的沟通将会有多好。为了做到这一点，我们需要知道人类多感官交流的基础是什么，这样我们才能确保机器能够在人类处理的必要容限内运行。这项研究使用了人工设备，如机器人下巴、有动画的会说话的头和会说话的手套，以及人类行为的扭曲，以探索交流过程在听觉和视觉通道中信号产生和处理的水平上所能承受的扭曲范围。言语行为中的结构性夸张(脸型)、功能性夸张(用手代替脸部)和行为夸张(响度)被用来描述听觉和视觉成分是如何相互对应和相互依赖的。我们把这种信号和结构之间的对应和相互依赖称为“一致性”。目标是确定在交流互动中断之前，人类处理系统可以承受多大的扭曲。研究这些扭曲对口语交流的产生和感知的影响将揭示什么在人机交互中起作用，并使我们更接近理解它是如何工作的。

Human-machine interaction is becoming more and more prevalent in our lives. Eventually, we predict that such interaction will be linguistically communicative and will entail machine emulation of a human conversant. Simple interfaces such as voice recognition over the telephone will be replaced by interaction with physical agents operating in multiple signal/sensory modalities. An obvious question is how well will humans and machines be able to communicate if their respective domains of behavior are matched to a fair degree. In order to do this, we need to know what the bases of multisensory communication are for humans, so that we can insure that machines can operate within the necessary tolerances of human processing. This research uses artificial devices such as a robot jaw, animated talking head, and a talking glove, along with distortions of human behavior to explore the range of distortions the communicative process can withstand at the levels of the production and processing of signals in auditory and visual modalities. Structural (face shape), functional (replacing the face with a hand), and behavioral (loudness) exaggerations in spoken behavior are used to characterize how the audible and visible components correspond and co-depend on one another. We call this correspondence and co-dependency between signals and structures "coherence". The goal is to determine how much distortion the human processing system can withstand before communicative interaction breaks down. Examining the effects of these distortions on both the production and perception of spoken communication will reveal what works in human-machine interaction and bring us closer to an understanding of how it works.