CRCNS Research Proposal: Learning by Looking: Modeling visual system representation formation via foveated sensing in a 3-D world

尽管最近取得了显着的进步，但当前的计算机视觉模型仍然像人类一样有效地理解物体的形状和物理世界的结构。这个项目的前提是人类感知和理解视觉世界的能力取决于积极的探索。通过构建一个模拟人类视觉系统关键方面的模型，并学会通过对其环境的积极探索来观察，该项目旨在扩大对人类视觉的理解，并评估模拟人类视觉的机器学习系统是否可以利用其优势。这一科学探索对认知神经科学和机器学习具有重要意义，并有可能为更强大和可靠的人工视觉系统的开发提供信息。为了增强对视觉的计算理解，弥合人类和机器感知之间的差距，该项目旨在开发一种生物学信息模型，该模型从视觉世界的主动探索中学习。目前的计算机视觉模型，受静态二维（2D）视图的均匀采样的约束，在表示三维（3D）环境结构方面存在不足。该项目测试的假设是，在一个结构化的三维环境中的主动传感将导致更强大的表示，更好地捕捉世界的几何形状，也更密切地对应于生物系统。 该项目的主要目标是：（1）开发一个具有独立的中央凹和外围子系统的模型，再加上对比中央凹和外围视图的对比学习方法，使系统能够在没有明确标签和不使用人工转换的情况下引导自己的学习;（2）将这种自我监督学习系统置于生动、高清的3D领域中，通过不同的采样策略促进主动学习;以及（3）使用计算机视觉基准和与人类神经（fMRI）响应的比对，彻底评估模型的学习表示是否相对于传统模型表现出增强的几何表示。通过这些目标，该项目旨在加深对人类如何学习看的理解，加强计算神经科学和机器学习之间的联系，并为更像人类，更强大和更可靠的人工视觉技术铺平道路。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估而被认为值得支持。

Despite remarkable recent strides, current computer vision models still grapple with understanding the shapes of objects and the structure of the physical world as effectively as humans do. This project begins from the premise that the human ability to perceive and understand the visual world hinges on active exploration. By constructing a model that emulates key aspects of the human visual system--and learns to see through active exploration of its environment--this project seeks to broaden the understanding of human vision and assess whether machine learning systems that emulate human vision can harness some of its strengths. This scientific exploration carries considerable implications for cognitive neuroscience and machine learning, and has potential to inform the development of more robust and reliable artificial vision systems.To enhance computational understanding of vision and bridge the gap between human and machine perception, this project aims to develop a biologically-informed model that learns from active exploration of the visual world. Current computer vision models, constrained by uniform sampling of static two dimensional (2D) views, fall short in representing a three dimensional (3D) environment structure. This project tests the hypothesis that active sensing in a structured three-dimensional environment will lead to more robust representations that better capture world geometry, and also correspond more closely to the biological system. The project's main objectives are: (1) to develop a model with separate foveal and peripheral subsystems, coupled with a contrastive learning approach that contrasts foveal and peripheral views, enabling the system to bootstrap its own learning without explicit labels and without using artificial transformations; (2) to place this self-supervised learning system within a vivid, high-definition 3D realm, promoting active learning via diverse sampling policies; and (3) to thoroughly assess whether the model's learned representations demonstrate enhanced geometric representation relative to conventional models, using both computer vision benchmarks and alignment with human neural (fMRI) responses. Through these aims, this project aims to deepen understanding of how humans learn to see, strengthen the ties between computational neuroscience and machine learning, and pave the way for more human-like, robust, and reliable artificial vision technologies.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.