NCS-FO: Neuroimaging to Advance Computer Vision, NLP, and AI

人们常说，一幅画胜过千言万语。通常，为了搜索需要的东西，无论是图像还是图像中的对象，都需要用文字来代替。为高效搜索获得准确的标签是计算机视觉的一个长期目标，但进展缓慢。这个项目采用了新的方法，通过利用最好的图像识别器——人类大脑，显著地改变了图像-词标签的完成方式。通过功能性磁共振成像和脑电图，记录人类观看图片/视频的大脑活动，然后用于提高涉及计算机视觉和自然语言处理的人工智能（AI）任务的性能。目前的系统使用机器学习来训练计算机识别图像/视频中的物体（名词）和活动（动词），然后用它们来描述事件。推理任务（例如，解决数学问题）就可以完成了。这些系统在专门准备的数据集上进行训练，这些数据集包含对象的名词、活动的动词、描述事件的句子以及考试问题和答案的样本。一种新的范例是在扫描人类大脑的同时让他们执行相同的任务，从而确定与这些任务相关的神经模式。大脑活动模式，反过来，被用来训练更好的计算机系统。核心假设是，理解人类对涉及预测的基础语言的处理及其在推理过程中的使用，将极大地改善工程计算机视觉、自然语言处理和执行图像/视频字幕、视觉问题回答和问题解决的人工智能系统。科学和工程目标包括开发与神经成像一致的人类语言基础和推理模型，以改进集成语言和视觉的工程系统，以支持自动推理。主要的科学问题是理解人类大脑识别、联系谓词和论证并将其用于推理的机制。这一假设，即视觉和语言表征中的谓词-论证联系是类似的，这将支持推理和解决问题，将使用多种神经成像模式和机器学习算法进行测试，以从处理语言和视觉刺激的受试者的大脑扫描中解码“谁对谁做了什么”。迭代方法将涉及理解神经层面的信息集成，通过训练计算机执行来自大规模自然任务的刺激的神经成像数据来执行日益复杂的任务，从而提高机器学习在人工智能任务上的性能。使用相同的人类和机器性能数据集将支持将科学进步转化为涉及计算机视觉和自然语言处理集成的工程实践。该奖项由国际科学与工程办公室共同资助。

It is often said that a picture is worth a thousand words. Frequently, to search for what is needed, whether images or objects in those images, words are needed instead. Getting accurate labels for efficient searches is a longstanding goal of computer vision, but progress has been slow. This project employs new methods to significantly change how picture-word labeling is accomplished by taking advantage of the best picture recognizer available, the human brain. Through functional magnetic resonance imaging and electroencephalography, brain activity of humans looking at pictures/videos is recorded and then used to improve performance on artificial intelligence (AI) tasks involving computer vision and natural language processing. Current systems use machine learning to train computers to recognize objects (nouns) and activities (verbs) in images/video, which are then used to describe events. Reasoning tasks (e.g., solving math problems) can then be done. These systems are trained on specially prepared datasets with samples of nouns for objects, verbs for activities, sentences describing events, and exam questions and answers. A novel paradigm using humans to perform the same tasks while their brains are scanned allows determination of neural patterns associated with those tasks. The brain activity patterns, in turn, are used to train better computer systems.The central hypothesis is that understanding human processing of grounded language involving predication and its use during reasoning will materially improve engineered computer vision, natural language processing, and AI systems that perform image/video captioning, visual question answering, and problem solving. Scientific and engineering goals include developing models of human language grounding and reasoning consistent with neuroimaging, to improve engineered systems integrating language and vision that support automated reasoning. The main scientific question is to understand mechanisms by which predicates and arguments are identified, linked, and used for reasoning by the human brain. The hypothesis, that predicate-argument linking in visual and linguistic representations are accomplished similarly, and that this then supports reasoning and problem solving, will be tested using multiple neuroimaging modalities, and machine learning algorithms to decode "who did what to whom" from brain scans of subjects processing linguistic and visual stimuli. The iterative approach will involve understanding information integration at the neural level, to improve machine learning performance on AI tasks by training computers to perform increasingly complex tasks with neuroimaging data from stimuli derived from large-scale natural tasks. Using identical datasets for human and machine performance will support translation of scientific advances to engineering practice involving integration of computer vision and natural language processing.This award is cofunded by the Office of International Science and Engineering.