CAREER: Defining how the primate visual system works under naturalistic conditions

日常生活中一些最重要的决定来自对视觉模式的识别。例如，医生在放射图像或显微镜载玻片中识别疾病迹象;驾驶员在能见度低的情况下检测道路上的障碍物;和/或保安人员筛选人员进入受限空间。灵长类动物的大脑是已知的最复杂的视觉模式识别系统，但对其算法的理解是有限的，部分原因是当神经元被测试时，它是有限的选择图像集，这些图像相对简单和明确。然而，简单的图像很难与自然图像联系起来，自然图像更加多样化和复杂。该项目将通过消除图像选择的限制，帮助发现神经元在视觉识别中的更广泛功能。该提案的新颖方法是让神经元与闭环系统中的机器智能模型“合作”，创建包含神经元编码的视觉信息的合成图像。这些合成图像代表了大脑评估为视觉世界中信息量最大的“样本”的模式-形状，颜色和纹理的独特组合，有助于识别物体，如面部或食物。这些样本是人工系统也应该存储的信息类型的线索。这些合成图像被称为“原型。“这个项目包括三个主要目标。这个新发明的闭环系统工作得非常好，所以很重要的是要理解为什么。第一个目标是解释用于图像生成的深度学习模型（“生成器”）通过检查其输入空间如何几何成形以创建自然图像来学习什么，以及神经元引导的搜索算法如何利用这个空间来创建原型。第二个目标是使用原型来预测神经元对自然图像的反应，这将通过创建距离函数来实现，该距离函数测量任何给定图像和任何给定原型之间的感知相似性。此外，这些距离函数还将用于将原型解构为更简单的视觉属性，如颜色和纹理。最后，第三个目标解释了这样一个事实，即神经元以集合的方式工作，以分布式的方式编码信息。图像合成方法将适用于从神经元群体中恢复视觉信息，而不仅仅是单个神经元。最后，除了为理解大脑中的视觉识别提供桥梁外，这些目标还将作为识别和打击伪造图像和视频的教育资源，即所谓的“DeepFakes”，这可能会破坏对媒体和新闻的信任。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估来支持。

Some of the most important decisions in everyday life come from the recognition of visual patterns. Example include doctors identifying signs of disease in radiology images or microscope slides; drivers detecting obstacles on the road under poor visibility conditions; and/or security guards screening people to enter restricted spaces. The primate brain is the most sophisticated visual pattern recognition system known, but understanding of its algorithms is limited in part because when neurons are tested, it is with limited sets of selected images that are relatively simple and unambiguous. Yet simple images are difficult to relate to natural images, which are more diverse and complex. This project will help discover the broader functions of neurons in visual recognition, by removing the constraint of image selection. This proposal's novel approach is to let neurons "team up" with machine intelligence models in a closed-loop system, creating synthetic images that contain visual information encoded by neurons. These synthetic images represent patterns that the brain has evaluated as being the most informative "samples" of the visual world -- unique combinations of shapes, colors, and textures useful in recognizing objects such as faces or food. These samples are clues for the kinds of information that artificial systems should also store. These synthetic images are called "prototypes."This project includes three major objectives. This newly invented closed-loop system works extremely well, so it is important to understand exactly why. The first objective is to explain what deep-learning models for image generation ("generators") learn by examining how their input space is shaped geometrically to create naturalistic images, and how this space is exploited by neuron-guided search algorithms to create prototypes. The second objective is to use prototypes to predict the responses of neurons to natural images, which will be done by creating distance functions that measure the perceptual similarity between any given image and any given prototype. Further, these distance functions will also serve to deconstruct prototypes into simpler visual attributes such as colors and textures. Finally, the third objective accounts for the fact that neurons work in ensembles, encoding information in a distributed way. The image synthesis approach will be adapted to recover visual information from neuronal populations, not just single neurons. Finally, beyond providing a bridge to understanding visual recognition in the brain, these objectives will serve as an educational resource for identifying and countering fabricated images and videos, so-called "DeepFakes," which can undermine trust in media and news.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.