CAREER: Defining how the primate visual system works under naturalistic conditions

点击翻译按钮获取中文摘要

Some of the most important decisions in everyday life come from the recognition of visual patterns. Example include doctors identifying signs of disease in radiology images or microscope slides; drivers detecting obstacles on the road under poor visibility conditions; and/or security guards screening people to enter restricted spaces. The primate brain is the most sophisticated visual pattern recognition system known, but understanding of its algorithms is limited in part because when neurons are tested, it is with limited sets of selected images that are relatively simple and unambiguous. Yet simple images are difficult to relate to natural images, which are more diverse and complex. This project will help discover the broader functions of neurons in visual recognition, by removing the constraint of image selection. This proposal's novel approach is to let neurons "team up" with machine intelligence models in a closed-loop system, creating synthetic images that contain visual information encoded by neurons. These synthetic images represent patterns that the brain has evaluated as being the most informative "samples" of the visual world -- unique combinations of shapes, colors, and textures useful in recognizing objects such as faces or food. These samples are clues for the kinds of information that artificial systems should also store. These synthetic images are called "prototypes."This project includes three major objectives. This newly invented closed-loop system works extremely well, so it is important to understand exactly why. The first objective is to explain what deep-learning models for image generation ("generators") learn by examining how their input space is shaped geometrically to create naturalistic images, and how this space is exploited by neuron-guided search algorithms to create prototypes. The second objective is to use prototypes to predict the responses of neurons to natural images, which will be done by creating distance functions that measure the perceptual similarity between any given image and any given prototype. Further, these distance functions will also serve to deconstruct prototypes into simpler visual attributes such as colors and textures. Finally, the third objective accounts for the fact that neurons work in ensembles, encoding information in a distributed way. The image synthesis approach will be adapted to recover visual information from neuronal populations, not just single neurons. Finally, beyond providing a bridge to understanding visual recognition in the brain, these objectives will serve as an educational resource for identifying and countering fabricated images and videos, so-called "DeepFakes," which can undermine trust in media and news.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.