Concurrency, synchrony, and interaction in multimodal communication

多模式通信中的并发、同步和交互

• 批准号: 261549-2007
• 负责人: VatikiotisBateson, Eric
• 金额: $ 2.91万
• 依托单位: University of British Columbia
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2007
• 资助国家: 加拿大
• 起止时间: 2007-01-01 至 2008-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=362341
• 关键词: Concurrency; synchrony; interaction; multimodal; communication

Human-machine interaction is becoming more and more prevalent in our lives. Eventually, we predict that such interaction will be linguistically communicative and will entail machine emulation of a human conversant. Simple interfaces such as voice recognition over the telephone will be replaced by interaction with physical agents operating in multiple signal/sensory modalities. An obvious question is how well will humans and machines be able to communicate if their respective domains of behavior are matched to a fair degree. In order to do this, we need to know what the bases of multisensory communication are for humans, so that we can insure that machines can operate within the necessary tolerances of human processing. This research uses  artificial devices such as a robot jaw, animated talking head, and a talking glove, along with distortions of human behavior to explore the range of distortions the communicative process can withstand at the levels of the production and processing of signals in auditory and visual modalities. Structural (face shape), functional (replacing the face with a hand), and behavioral (loudness) exaggerations in spoken behavior are used to characterize how the audible and visible components correspond and co-depend on one another. We call this correspondence and co-dependency between signals and structures "coherence". The goal is to determine how much distortion the human processing system can withstand before communicative interaction breaks down. Examining the effects of these distortions on both the production and perception of spoken communication will reveal what works in human-machine interaction and bring us closer to an understanding of how it works.

人机交互在我们的生活中变得越来越普遍。最终，我们预测这种交互将在语言上进行交流，并且需要机器模拟人类熟悉的人。电话语音识别等简单界面将被以多种信号/感知模式运行的物理代理交互所取代。一个明显的问题是，如果人类和机器各自的行为领域达到相当程度的匹配，那么他们的沟通能力如何。为了做到这一点，我们需要知道人类多感官交流的基础是什么，以便我们可以确保机器可以在人类处理的必要容限内运行。这项研究使用机器人下巴、会说话的动画头和会说话的手套等人造设备，以及人类行为的扭曲来探索交流过程在听觉和视觉方式的信号产生和处理层面上可以承受的扭曲范围。言语行为中的结构（面部形状）、功能（用手代替面部）和行为（响度）夸张用于描述听觉和视觉成分如何相互对应和相互依赖。我们将信号和结构之间的这种对应和相互依赖称为“一致性”。目标是确定人类处理系统在交流互动崩溃之前可以承受多少失真。检查这些扭曲对口头交流的产生和感知的影响将揭示人机交互的原理，并使我们更接近于理解它是如何工作的。