Real-Time Multisensory Integration for Time-Varying Signals

描述(申请人提供)：传统的感知模型有一个基本原则，即感觉处理是分层的：感觉过滤器将物理世界分解为原始特征，这些原始特征通过一系列阶段重新组合成连贯的超通道表示。这些模型通常被概念化为处理时不变(即，恒定)信号或表示，并在其结构(基线-刺激-反应)与假定的潜在处理动态的离散和阶段性平行的实验中进行评估。真实环境包含连续的时变信号，需要连续的感知和行为解决方案，虽然假设时不变模型的操作将平凡地概括到连续的时间域，但有理由怀疑它们不会。例如，在连续的动态环境中，输入轨迹的波动可被解释为表示噪声污染或源信号的变化。大脑如何在这种情况下获得稳定的知觉解决方案是至关重要的。本申请描述了一系列 寻求在多感官整合的背景下评估这一问题的短期实验。跨感官信息的融合已被证明可以改善感知和行为判断，并加快对外部事件的反应，特别是那些其单一感觉表征被噪声严重污染或掩盖的事件。为了实现这些好处，大脑必须协调具有不同操作参数的感官之间的活动，特别是依赖于直接环境环境的不同可靠性。为了实现最佳整合，大脑必须在联合考虑时，根据这些可靠性对来自不同感官的信号进行适当的加权。这里提出的框架试图通过评估受试者是否使用卡尔曼滤波动力学来实现连续时间多感觉任务的最佳性能来理解这一现象。受试者的任务是跟踪单独或协同呈现的动态和受噪声破坏的视觉和听觉刺激模式。受试者对信号变化的适应能力和感觉的相对可靠性在拟议的模型框架中进行了解释。受试者的预期是由信号和信号统计变化的先前经验和预警形成的，并评估他们根据该信息在回应中做出调整的时机和影响。所提出的研究中采用的实验方法将对感官通道的组合原理提供重要的见解，并有助于建立实现实时整合的边界条件。

DESCRIPTION (provided by applicant): Traditional models of perception have a foundational principle that sensory processing is hierarchical: sensory filters decompose the physical world into primitive features that are recombined through a series of stages into a coherent supramodal representation. These models are typically conceptualized as processing time-invariant (i.e., constant) signals or representations, and are evaluated in experiments whose structure (baseline->stimulus->response) parallels the discrete and staged nature of the presumed underlying processing dynamics. Real environments contain continuous time-varying signals and demand continuous perceptual and behavioral solutions, and while it is assumed that the operations of time-invariant models will trivially generalize to the continuous time domain, there are reasons to suspect that they will not. For example, in a continuous, dynamic environment, fluctuations in input traces may be interpreted as either representing noise contamination or changes in the source signals. How the brain achieves stable perceptual solutions in such circumstances is of critical interest. The present application describes a series of short-term experiments that seek to evaluate this issue in the context of multisensory integration. The merging of information across the senses has been shown to improve perceptual and behavioral judgments and speed the responses to external events, particularly those whose unisensory representations are significantly contaminated or obscured by noise. To realize these benefits, the brain must coordinate activity across senses that have different operational parameters, in particular, different reliabilities that are dependent on the immediate environmental circumstances. To achieve optimal integration, the brain must appropriately "weight" signals derived from different senses according to these reliabilities in their joint consideration. The framework proposed here seeks to understand this phenomenon by evaluating whether subjects use Kalman filter dynamics to achieve optimal performance on a continuous-time multisensory task. Subjects are tasked with tracking dynamic and noise-corrupted patterns of visual and auditory stimuli presented alone or in concert. The ability of subjects to adapt to changes in the signal and the relative reliabilities of the senses are interpreted in the context of the proposed model framework. Subject expectations are shaped by prior experience and forewarning of changes in the signals and signal statistics, and the timing and impact of the adjustments they make in their responses consequent to this information is evaluated. The experimental approach applied in the proposed studies will provide important insights into the principles upon which sensory channels are combined and help to establish the boundary conditions upon which real-time integration is achieved.