US-German Collaboration: Towards a Neural Theory of 3D Shape Perception

大脑如何估计我们周围物体的三维形状仍然是视觉神经科学中最重要的挑战之一。视网膜提供的信息基本上是模糊的，因为3D形状、光照和表面反射率的许多不同组合与任何给定的图像是一致的。尽管存在这种模糊性，但视觉系统非常擅长在广泛的观看条件下估计3D形状，这是现有机器视觉系统无法做到的。该项目的长期目标是开发一个神经术语的计算模型，以解释灵长类动物视觉系统如何估计3D形状。它将建立在视觉皮层（V1）早期细胞的反应基础上，并开发出它们如何被组织成指定3D形状属性的中级配置的模型。重要的是，该项目还将在一系列旨在测试特定预测的心理物理实验中测量人类对3D形状的感知，将Roland W. Fleming（吉森大学：人类感知，心理物理学）和Steven W. Zucker（耶鲁大学：计算视觉，计算神经科学）的互补专业知识汇集在一起。这一结果将对腹侧脑加工流的视觉回路特性提供更深入的理解；他们应该提供3D计算机视觉和图形模型；它们可能为视力缺陷患者的康复策略的发展铺平道路。基本的方法是从调整到不同方向的神经元群开始，并试图理解这些神经元如何根据微分几何原理提供有关局部形状属性的基本信息。具体来说，当3D表面投射到视网膜上时，阴影和纹理的扭曲梯度导致局部图像方向或方向场的高度结构化模式，这可以通过涉及远程水平连接的电路推断出来。研究人员试图推导出正式的模型，展示如何组织这些网络来推断3D表面特性。具体方法包括四个阶段：(i)建模视觉系统如何通过横向相互作用和反馈从模型V1细胞的输出中获得干净可靠的方向场；（ii）建立如何将局部测量分组到特定的“中级”配置中，以支持3D形状属性的恢复（建模V2到V4）；（iii）建模如何将这些低级和中级2D测量值映射为3D形状属性的表示（V4到IT）；（iv）建模分组和全局约束如何将这些形状估计转换为全局形状重建（同样是V4到IT）。有针对性的心理物理实验将补充所有的建模和测试具体的预测。由此产生的刺激将支持下一代神经生理学实验。尽管上述阶段定义了一个工作策略，但这些阶段之间的依赖关系也应该提供一个连接不同皮层区域的前馈/反馈投影模型。最终的目标是建立一个能够正确预测人类形状感知的错误、成功和限制的模型。该项目由计算神经科学合作研究和国际科学与工程办公室联合资助。德国教育和研究部（BMBF）正在资助一个伙伴项目。

How the brain estimates the 3D shape of objects in our surroundings remains one of the most significant challenges in visual neuroscience. The information provided by the retina is fundamentally ambiguous, because many different combinations of 3D shape, illumination and surface reflectance are consistent with any given image. Despite this ambiguity, the visual system is extremely adept at estimating 3D shape across a wide range of viewing conditions, something that no extant machine vision system can do. The long-term goal of the project is to develop a computational model in neural terms to explain how 3D shape is estimated in the primate visual system. It will build upon the responses of cells early in visual cortex (V1) and develop models of how they can be organized into mid-level configurations that specify 3D shape properties. Importantly, the project will also measure human perception of 3D shape in a series of psychophysical experiments designed to test specific predictions, bringing together the complementary expertise of Roland W. Fleming (Giessen University: human perception, psychophysics) and Steven W. Zucker (Yale University: computational vision, computational neuroscience). The results should provide a deeper understanding of visual circuit properties in the ventral processing stream; they should provide models for 3D computer vision and graphics; and they may pave the way for the development of rehabilitation strategies for patients with visual deficits.The basic approach starts with populations of neurons tuned to different orientations and seeks to understand how these provide basic information about local shape properties according to the principles of differential geometry. Specifically, when 3D surfaces are projected onto the retina, the distorted gradients of shading and texture lead to highly structured patterns of local image orientation, or orientation fields, which can be inferred via circuits involving long-range horizontal connections. The investigators seek to derive formal models showing how these networks can be organized to infer 3D surface properties. The specific approach is involves four stages: (i) modeling how the visual system obtains clean and reliable orientation fields from the outputs of model V1 cells through lateral interactions and feedback; (ii) establishing how local measurements are grouped into specific "mid-level" configurations to support the recovery of 3D shape properties (modeling V2 to V4); (iii) modeling how these low- and mid-level 2D measurements can be mapped into representations of 3D shape properties (V4 to IT); and (iv) modeling how grouping and global constraints can convert these shape estimates into global shape reconstructions (again V4 to IT). Targeted psychophysical experiments will complement all of the modeling and test specific predictions from it. The resulting stimuli will support next generation neurophysiological experiments. Although the above stages define a working strategy, dependencies among these stages should also provide a model of the feedforward/feedback projections that link different areas of cortex. The ultimate goal is a model that can correctly predict the errors, the successes, and the limits of human shape perception. This project is jointly funded by Collaborative Research in Computational Neuroscience and the Office of International Science and Engineering. A companion project is being funded by the German Ministry of Education and Research (BMBF).