Concurrency, synchrony, and interaction in multimodal communication

人机交互在我们的生活中变得越来越普遍。最终，我们预测这种互动将是语言交流的，并将需要机器模拟人类熟悉。简单的界面，如通过电话进行语音识别，将被以多种信号/感觉方式操作的物理代理的交互所取代。一个明显的问题是，如果人类和机器各自的行为领域在相当程度上匹配，那么它们能够进行多好的沟通。为了做到这一点，我们需要知道人类多感官交流的基础是什么，这样我们就可以确保机器可以在人类处理的必要公差范围内运行。本研究使用人工设备，如机器人下巴，动画会说话的头，和一个会说话的手套，沿着人类行为的扭曲，以探讨失真的范围内的沟通过程中可以承受的生产和处理的听觉和视觉形式的信号的水平。口语行为中的结构（面部形状）、功能（用手代替面部）和行为（响度）夸张被用来表征听觉和视觉组件如何相互对应和相互依赖。我们把信号和结构之间的这种对应和相互依赖称为“连贯性”。其目的是确定在交流互动中断之前，人类处理系统可以承受多少失真。研究这些扭曲对口语交流的产生和感知的影响，将揭示人机交互中的工作原理，并使我们更接近于理解它是如何工作的。

Human-machine interaction is becoming more and more prevalent in our lives. Eventually, we predict that such interaction will be linguistically communicative and will entail machine emulation of a human conversant. Simple interfaces such as voice recognition over the telephone will be replaced by interaction with physical agents operating in multiple signal/sensory modalities. An obvious question is how well will humans and machines be able to communicate if their respective domains of behavior are matched to a fair degree. In order to do this, we need to know what the bases of multisensory communication are for humans, so that we can insure that machines can operate within the necessary tolerances of human processing. This research uses artificial devices such as a robot jaw, animated talking head, and a talking glove, along with distortions of human behavior to explore the range of distortions the communicative process can withstand at the levels of the production and processing of signals in auditory and visual modalities. Structural (face shape), functional (replacing the face with a hand), and behavioral (loudness) exaggerations in spoken behavior are used to characterize how the audible and visible components correspond and co-depend on one another. We call this correspondence and co-dependency between signals and structures "coherence". The goal is to determine how much distortion the human processing system can withstand before communicative interaction breaks down. Examining the effects of these distortions on both the production and perception of spoken communication will reveal what works in human-machine interaction and bring us closer to an understanding of how it works.