Understanding feature-based auditory-visual interactions.

描述(由申请者提供)：研究揭示了视觉系统的许多机制。然而，知觉体验通常是多通道的，视觉和听觉通道之间有着密切的关系。听觉信号影响整个视觉通路的神经激活，包括中脑和初级视觉皮质。因此，重要的是要扩展严格的视觉理论，以整合多模式背景。以往对听觉-视觉交互作用的研究主要集中在对空间、时间、持续时间、运动和语音的感知上，而最近的研究已经证明了听觉-视觉交互作用在感知物体和面部方面的作用。本研究的目的是填补我们在视觉特征加工水平上对视听交互作用的理解的空白。我们将表征哪些声学模式与低级(例如，空间频率)、中级(例如，材料纹理和2D形状)和高级(例如，常见对象、单词、面部识别和面部表情)视觉特征的处理唯一地交互作用。为了了解这些相互作用，我们将结合心理物理学和计算模型(目标1)来确定相关声音如何影响视觉特征处理的基本机制，包括控制图像可见性(前端信噪比和采样效率)的机制，控制视觉意识的信号竞争的机制，以及在存在接受野之间和接受野内信号交互的情况下控制视觉特征的神经群体编码的强度和可靠性的机制。这些结果将提供对声音如何影响视觉信号、采样、竞争和编码的综合理解，以处理低级别、中级和高级视觉特征。这项拟议的研究还将允许开发跨模式方法，通过增强特定的空间尺度、材料、形状、物体和面部表情来帮助视觉感知。例如，我们的初步结果表明，声音可以用来增强和调节对面部表情的感知，并将注意力引导到特定的空间频率。在翻译目标(AIM 2)中，我们将系统地研究如何使用声音来帮助视觉感知，例如，将注意力引导到搜索中的对象、材料、单词或面部表情，通过将注意力引导到诊断的空间频率分量来促进对象识别，以及通过将注意力引导到多个空间尺度来丰富场景理解。因为特征特定的听觉信号很容易通过耳机呈现，所以所提出的研究可以提供一种手段来例如对抗偏向知觉(例如，由于社交焦虑而将面部表情感知为负面)，并针对具有视觉挑战的个人，例如低视力、影响视力的中风或具有注意力障碍的个人，将注意力引导到特定对象和空间尺度(例如，细节与主旨)。因此，所提出的研究不仅将系统地将听觉影响整合到现有的视觉特征处理模型中，而且还可能提供一种利用听觉信号辅助视觉处理的方法。

DESCRIPTION (provided by applicant): Research has revealed much about the mechanisms of the visual system. However, perceptual experience is usually multimodal, with close relationships between visual and auditory modalities. Auditory signals influence neural activation throughout the visual pathways including in the mid brain and primary visual cortex. It is therefore important to extend the rigorous theories of vision to integrate multimodal contexts. Prior research on auditory-visual interactions has primarily focused on perception of space, timing, duration, motion, and speech, whereas recent research has demonstrated auditory-visual interactions in the perception of objects and faces. The goal of the proposed research is to fill the gap in our understanding of auditory-visual interactions at the level of visual feature processing. We will characterize which acoustic patterns uniquely interact with processing of low-level (e.g., spatial frequency), intermediate-level (e.g., material texture and 2D shape), and high-level (e.g., common objects, words, face identity, and facial expressions) visual features. To understand these interactions, we will combine psychophysics and computational modeling (AIM 1) to determine how associated sounds influence basic mechanisms of visual feature processing, including those that control image visibility (front-end signal-to-noise ratio and sampling efficiency), those that control signal competition for visual awareness, and those that control the strength and reliability of neural population coding of visual features in the presence of between- and within-receptive-field signal interactions. The results will provide an integrative understanding of how sounds influence visual signals, sampling, competition, and coding, for the processing of low-, intermediate-, and high-level visual features. The proposed research will also allow development of cross- modal methods for assisting visual perception by enhancing specific spatial scales, materials, shapes, objects, and facial expressions. For example, our preliminary results suggest that sounds can be used to boost and tune the perception of facial expressions, and to direct attention to specific spatial frequencies. In the translational aim (AIM 2), we will systematically investigate how sounds can be used to aid visual perception, for example, to direct attention to an object, material, word, or facial expression in search, facilitate object recognition via directing attention to diagnostic spatial-frequency components, and enrich scene understanding via directing attention to multiple spatial scales. Because feature-specific auditory signals are readily presented over headphones, the proposed research may provide a means to, for example, counter biased perception (e.g., perceiving facial expressions as negative due to social anxiety), and to direct attention to specific objects and spatial scales (e.g., details versus gist) for individuals with visual challenges such as low vision, strokes affecting vision, or with attention disorders. Thus, the proposed research will not only systematically integrate auditory influences into the current models of visual feature processing, but it may also provide a means to aid visual processing by using auditory signals.