The natural statistics of audiovisual speech.
The natural statistics of audiovisual speech.
复制标题
DOI：
10.1371/journal.pcbi.1000436
复制
发表时间：
2009-07
影响因子：
4.3
通讯作者：
Ghazanfar AA
中科院分区：
文献类型：
--
作者：
Chandrasekaran C;Trubanova A;Stillittano S;Caplier A;Ghazanfar AA
人类和其他动物一样，暴露在连续的信号流中，这些信号在本质上是动态的、多模态的、扩展的和时变的。这个复杂的输入空间必须由我们的感觉系统转换和采样，并传输到大脑，在那里它可以指导选择适当的行动。为了简化这一过程，有人认为大脑利用了刺激空间中的统计信息。对这一想法的测试主要局限于单峰信号和自然场景。一个重要的类别的多感官信号的定量输入空间表征是不可用的是人类的语音。我们不明白我们的大脑必须从视听语音流中主动拼凑哪些信号才能获得感知，而不是已经嵌入流本身的信号结构中的信号。从本质上讲，我们对视听语音的自然统计没有明确的认识。在本研究中，我们确定了视听语音的以下主要统计特征。首先，我们观察到强大的相关性和密切的时间对应关系之间的地区开口和声包络。第二，我们发现张口面积和声道共鸣之间的相关性最强。第三，我们观察到，在2-7 Hz的频率范围内，嘴巴张开的区域和语音包络都是时间调制的。最后，我们表明，嘴部运动的时间相对于发病的声音是一致的100和300毫秒之间。我们解释这些数据的背景下，最近的神经理论的讲话，这表明，语音通信是一个耦合，多感官事件，从而输出的信号是匹配的接收器的神经过程。当我们看着别人说话时，我们的大脑实际上做了多少工作？言语结构本身在多大程度上促进了这一工作？我们的工作表明，不仅语音的视觉和听觉成分被紧紧锁定（避免了大脑主动绑定这些信息的需要），这种时间协调也具有明显的节奏，在2到7 Hz之间。此外，在语音产生期间，声音的开始相对于嘴的初始可见运动以100和300 ms之间的延迟发生。这些视听语音的时间参数是有趣的，因为它们符合听觉皮层神经元振荡的已知特性。因此，考虑到我们已经知道的关于语音的神经处理，视听语音信号的自然特征似乎是最佳结构，以便与接收者正在进行的大脑节律相互作用。
Humans, like other animals, are exposed to a continuous stream of signals, which are dynamic, multimodal, extended, and time varying in nature. This complex input space must be transduced and sampled by our sensory systems and transmitted to the brain where it can guide the selection of appropriate actions. To simplify this process, it's been suggested that the brain exploits statistical regularities in the stimulus space. Tests of this idea have largely been confined to unimodal signals and natural scenes. One important class of multisensory signals for which a quantitative input space characterization is unavailable is human speech. We do not understand what signals our brain has to actively piece together from an audiovisual speech stream to arrive at a percept versus what is already embedded in the signal structure of the stream itself. In essence, we do not have a clear understanding of the natural statistics of audiovisual speech. In the present study, we identified the following major statistical features of audiovisual speech. First, we observed robust correlations and close temporal correspondence between the area of the mouth opening and the acoustic envelope. Second, we found the strongest correlation between the area of the mouth opening and vocal tract resonances. Third, we observed that both area of the mouth opening and the voice envelope are temporally modulated in the 2–7 Hz frequency range. Finally, we show that the timing of mouth movements relative to the onset of the voice is consistently between 100 and 300 ms. We interpret these data in the context of recent neural theories of speech which suggest that speech communication is a reciprocally coupled, multisensory event, whereby the outputs of the signaler are matched to the neural processes of the receiver. When we watch someone speak, how much work is our brain actually doing? How much of this work is facilitated by the structure of speech itself? Our work shows that not only are the visual and auditory components of speech tightly locked (obviating the need for the brain to actively bind such information), this temporal coordination also has a distinct rhythm that is between 2 and 7 Hz. Furthermore, during speech production, the onset of the voice occurs with a delay of between 100 and 300 ms relative to the initial, visible movements of the mouth. These temporal parameters of audiovisual speech are intriguing because they match known properties of neuronal oscillations in the auditory cortex. Thus, given what we already know about the neural processing of speech, the natural features of audiovisual speech signals seem to be optimally structured for their interactions with ongoing brain rhythms in receivers.
登录
查看更多内容
影响因子：
4.3
作者：
Kiebel SJ;Daunizeau J;Friston KJ
通讯作者：
Friston KJ
影响因子：
--
作者：
HIRSH, IJ;SHERRICK, CE
通讯作者：
SHERRICK, CE
影响因子：
1.9
作者：
Greenberg, S;Carvey, H;Chang, SY
通讯作者：
Chang, SY
DOI：
10.1098/rstb.2007.2155
发表时间：
2008-03-12
影响因子：
6.3
作者：
Campbell, Ruth
通讯作者：
Campbell, Ruth
影响因子：
2.4
作者：
Cooke, Martin;Barker, Jon;Shao, Xu
通讯作者：
Shao, Xu