CRCNS US-France Research Proposal: Oscillatory processes for visual reasoning in deep neural networks

近年来，深度卷积网络(DCNS)的发展在机器视觉领域取得了巨大的成功。尽管取得了这些成功，但到目前为止，在图像分类任务中获得了最令人印象深刻的结果，例如指示图像是否包含特定对象。然而，DCNS解决更复杂的视觉推理问题的能力仍然有限，例如理解对象之间的视觉关系。有趣的是，目前计算机视觉的许多工作都致力于扩展DCNS，但这些模型仍然被大脑的能力和多功能性所击败，也许部分原因是大脑回路可以进行更丰富的神经元计算。挑战是确定哪些神经机制是相关的，并找到合适的抽象来对它们进行建模。一组有希望的候选者是在大脑中发现的神经振荡。该项目旨在识别关键的振荡成分并表征人类解决视觉推理任务的能力的神经计算，并在现代深度学习架构中使用类似的策略。该项目将使用现有的计算模型来开发用于脑电研究的任务和刺激，以确定人类视觉推理能力的关键振荡成分。对这些脑电数据的分析将以生物物理-现实计算神经科学模型的发展为指导。这将为关于振荡簇背后的电路机制的假说的发展提供信息，并将这些机制与神经计算联系起来。最后，该项目将开发这些神经计算的新型机器学习理想化，这些理想化可以用当前的深度学习方法进行训练，但仍然可以在神经电路级别上解释。特别是，该项目将进一步开发基于复值神经元单元的振荡的初始机器学习公式，从而扩展该方法并展示其定性捕获视觉推理背后的关键振荡过程的能力。法国国家研究机构(ANR)正在资助一个配套项目。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

The development of deep convolutional networks (DCNs) has recently led to great successes in machine vision. Despite these successes, to date, the most impressive results have been obtained for image categorization tasks such as indicating whether an image contains a particular object. However, DCNs ability to solve more complex visual reasoning problems such as understanding the visual relations between objects remains limited. Interestingly, much work in computer vision is currently being devoted to extending DCNs, but these models are still outmatched by the power and versatility of the brain, perhaps in part due to the richer neuronal computations available to cortical circuits. The challenge is to identify which neuronal mechanisms are relevant and to find suitable abstractions to model them. One promising set of candidates is the neural oscillations that are found throughout the brain. This project seeks to identify the key oscillatory components and characterize the neural computations underlying humans ability to solve visual reasoning tasks, and to use similar strategies in modern deep learning architectures.This project will use existing computational models to develop tasks and stimuli to be used in EEG studies to identify the key oscillatory components underlying human visual reasoning ability. The analysis of these EEG data will be guided by the development of a biophysically-realistic computational neuroscience model. This will inform the development of hypotheses on the circuit mechanisms underlying the oscillatory clusters and relate these mechanisms to neural computations. Finally, the project will develop novel machine learning idealizations of these neural computations, which are trainable with current deep learning methods but still interpretable at the neural circuit level. In particular, the project will further develop initial machine learning formulation of oscillations based on complex-valued neuronal units, thus extending the approach and demonstrating its ability to qualitatively capture key oscillatory processes underlying visual reasoning. A companion project is being funded by the French National Research Agency (ANR).This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.