Collaborative Research: Neural and computational models of spatio-temporally varying natural scenes

当我们在我们的视觉环境中移动时，进入我们眼睛的光的模式强烈地受到环境中物体的特性、它们相对于彼此的运动以及我们自身相对于外部世界的运动的影响。这个合作项目将量化自然场景中的运动，记录早期视觉通路中神经元群体对运动的反应活动，并开发跨神经元群体的运动表示模型。这项工作的主要目标是在视觉处理的早期阶段充分描述自然场景中运动的生物表征，为对视觉感知至关重要的皮质计算奠定基础，并将生物学发现与计算机视觉社区的运动计算模型统一起来。视觉运动的感知对于生物系统和计算机视觉系统都是至关重要的。运动揭示了世界的结构，包括物体的相对和绝对深度，物体之间的表面边界，以及关于自我运动和其他物体的独立运动的信息。视觉运动对自然视觉场景的空间局部性和全局性之间的关系的影响，以及大脑早期视觉通路是如何表现这一点的，在很大程度上是未知的。该项目利用分布式神经系统和计算机视觉算法，使用一组新的复杂自然刺激来计算自然视觉场景的局部和全局属性，其中场景的地面真实属性是已知的，场景的所有方面，包括其反射率、表面属性、照明和运动都在研究人员的控制之下。将采用统一的概率建模框架，将自然场景属性的计算模型和生物模型联系在一起。神经活动将从大量密集采样的单个神经元中记录下来，这些神经元来自视觉丘脑。从计算机视觉领域的角度来看，从2D图像序列推断外部环境(或光流)的运动是一个重要的挑战。从神经科学界的角度来看，量化早期视觉通路中亮度和运动的分布式神经表征将是理解场景信息是如何被提取并准备在高级视觉中心进行处理的关键步骤。一组在计算机科学、工程和神经科学方面有经验的研究人员将开发一套理论基础和丰富的方法，用于用大脑和机器来表示和恢复局部亮度、局部运动边界和全局运动。

As we move through our visual environment, the pattern of light that enters our eyes is strongly shaped by the properties of objects within the environment, their motion relative to each other, and our own motion relative to the external world. This collaborative project will quantify motion within natural scenes, record activity from populations of neurons in the early visual pathway in response to the motion, and develop models of motion representation across neuronal populations. The primary goals of the work are to fully characterize the biological representation of motion in natural scenes in the early stages of visual processing that sets the stage for cortical computation critical for visual perception, and to unify the biological findings with computational models of motion from the computer vision community. The perception of visual motion is critical for both biological and computer vision systems. Motion reveals structure of the world including the relative and absolute depths of objects, surface boundaries between objects and information about ego-motion and the independent motion of other objects. The effects of visual motion on the relationship between spatially localized and global properties of the natural visual scene, and how this is represented by the early visual pathway of the brain, are largely unknown. This project addresses the computation of local and global properties of natural visual scenes by both distributed neural systems and computer vision algorithms using a novel set of complex naturalistic stimuli in which ground truth properties of the scene are known, and all aspects of the scene, including its reflectance, surface properties, lighting and motion are under investigator control. A unified probabilistic modeling framework will be adopted, that ties together the computational and biological models of properties of the natural scene. Neural activity will be recorded from a large population of densely sampled single neurons from the visual thalamus. From the perspective of the computer vision community, an important challenge exists in inferring the motion of the external environment (or "optical flow") from sequences of 2D images. From the perspective of the neuroscience community, quantifying the distributed neural representation of luminance and motion in the early visual pathway will be a critical step in understanding how scene information is extracted and prepared for processing in higher visual centers. A team of investigators with experience in computer science, engineering, and neuroscience will develop a theoretical foundation and rich set of methods for the representation and recovery of local luminance, local motion boundaries and global motion by brains and machines.