权益分类	功能权益	普通用户	{{item.name}}会员
{{category.name}}	{{benefitItem.name}}

NCS-FO: Neuroimaging to Advance Computer Vision, NLP, and AI

NCS-FO：神经影像学促进计算机视觉、NLP 和 AI

基本信息

批准号：
1734938
负责人：
Jeffrey Siskind
金额：
$ 100万
依托单位：
Purdue University
依托单位国家：
美国
项目类别：
Standard Grant
财政年份：
2017
资助国家：
美国
起止时间：
2017-08-15 至 2023-09-30
项目状态：
已结题

来源：
https://www.nsf.gov/awardsearch/showAward?AWD_ID=1734938&HistoricalAwards=false
关键词：
NCS FO Neuroimaging Advance Computer

项目摘要

It is often said that a picture is worth a thousand words. Frequently, to search for what is needed, whether images or objects in those images, words are needed instead. Getting accurate labels for efficient searches is a longstanding goal of computer vision, but progress has been slow. This project employs new methods to significantly change how picture-word labeling is accomplished by taking advantage of the best picture recognizer available, the human brain. Through functional magnetic resonance imaging and electroencephalography, brain activity of humans looking at pictures/videos is recorded and then used to improve performance on artificial intelligence (AI) tasks involving computer vision and natural language processing. Current systems use machine learning to train computers to recognize objects (nouns) and activities (verbs) in images/video, which are then used to describe events. Reasoning tasks (e.g., solving math problems) can then be done. These systems are trained on specially prepared datasets with samples of nouns for objects, verbs for activities, sentences describing events, and exam questions and answers. A novel paradigm using humans to perform the same tasks while their brains are scanned allows determination of neural patterns associated with those tasks. The brain activity patterns, in turn, are used to train better computer systems.The central hypothesis is that understanding human processing of grounded language involving predication and its use during reasoning will materially improve engineered computer vision, natural language processing, and AI systems that perform image/video captioning, visual question answering, and problem solving. Scientific and engineering goals include developing models of human language grounding and reasoning consistent with neuroimaging, to improve engineered systems integrating language and vision that support automated reasoning. The main scientific question is to understand mechanisms by which predicates and arguments are identified, linked, and used for reasoning by the human brain. The hypothesis, that predicate-argument linking in visual and linguistic representations are accomplished similarly, and that this then supports reasoning and problem solving, will be tested using multiple neuroimaging modalities, and machine learning algorithms to decode "who did what to whom" from brain scans of subjects processing linguistic and visual stimuli. The iterative approach will involve understanding information integration at the neural level, to improve machine learning performance on AI tasks by training computers to perform increasingly complex tasks with neuroimaging data from stimuli derived from large-scale natural tasks. Using identical datasets for human and machine performance will support translation of scientific advances to engineering practice involving integration of computer vision and natural language processing.This award is cofunded by the Office of International Science and Engineering.

人们常说，一张图片胜过千言万语。通常，为了搜索所需的内容，无论是图像还是图像中的对象，都需要文字。获得准确的标签以进行有效的搜索是计算机视觉的长期目标，但进展缓慢。该项目采用新的方法，通过利用最好的图片识别器，人类大脑，显着改变如何完成图片-单词标记。通过功能性磁共振成像和脑电图，人类观看图片/视频的大脑活动被记录下来，然后用于提高涉及计算机视觉和自然语言处理的人工智能（AI）任务的性能。当前的系统使用机器学习来训练计算机识别图像/视频中的对象（名词）和活动（动词），然后用于描述事件。推理任务（例如，解决数学问题），然后可以完成。这些系统在专门准备的数据集上进行训练，这些数据集包含对象的名词，活动的动词，描述事件的句子以及考试问题和答案。一种新的范式使用人类执行相同的任务，同时扫描他们的大脑，可以确定与这些任务相关的神经模式。大脑活动模式反过来又被用来训练更好的计算机系统。核心假设是，理解人类对涉及预测的基础语言的处理及其在推理过程中的使用，将大大改善工程计算机视觉、自然语言处理以及执行图像/视频字幕、视觉问答和解决问题的人工智能系统。科学和工程目标包括开发与神经成像一致的人类语言基础和推理模型，以改进集成语言和视觉的工程系统，支持自动推理。主要的科学问题是理解谓词和论点被识别、联系并被人脑用于推理的机制。这个假设，即视觉和语言表征中的谓词-论元链接是类似完成的，然后支持推理和解决问题，将使用多种神经成像方式和机器学习算法进行测试，以从处理语言和视觉刺激的受试者的大脑扫描中解码“谁对谁做了什么”。迭代方法将涉及在神经水平上理解信息集成，通过训练计算机使用来自大规模自然任务的刺激的神经成像数据执行越来越复杂的任务，来提高人工智能任务的机器学习性能。使用人类和机器性能的相同数据集将支持将科学进步转化为工程实践，涉及计算机视觉和自然语言处理的集成。该奖项由国际科学与工程办公室共同资助。

项目成果

期刊论文数量（20）

专著数量（0）

科研奖励数量（0）

会议论文数量（0）

专利数量（0）

Age of acquisition effects differ across linguistic domains in sign language: EEG evidence

手语中不同语言领域的习得年龄效应有所不同：脑电图证据

DOI：
10.1016/j.bandl.2019.104708
发表时间：
2020
期刊：
Brain and Language
影响因子：
2.5
作者：
Malaia, Evie A.;Krebs, Julia;Roehm, Dietmar;Wilbur, Ronnie B.
通讯作者：
Wilbur, Ronnie B.

Divide-and-conquer checkpointing for arbitrary programs with no user annotation

DOI：
10.1080/10556788.2018.1459621
发表时间：
2017-08
期刊：
Optimization Methods and Software
影响因子：
2.2
作者：
J. Siskind;Barak A. Pearlmutter
通讯作者：
J. Siskind;Barak A. Pearlmutter

Psycholinguistic mechanisms of classifier processing in sign language.

DOI：
10.1037/xlm0000958
发表时间：
2021-06
期刊：
Journal of experimental psychology. Learning, memory, and cognition
影响因子：
0
作者：
Krebs J;Malaia E;Wilbur RB;Roehm D
通讯作者：
Roehm D

Perturbation confusion in forward automatic differentiation of higher-order functions

DOI：
10.1017/s095679681900008x
发表时间：
2012-11
期刊：
Journal of Functional Programming
影响因子：
1.1
作者：
Oleksandr Manzyuk;Barak A. Pearlmutter;Alexey Radul;David R. Rush;J. Siskind
通讯作者：
Oleksandr Manzyuk;Barak A. Pearlmutter;Alexey Radul;David R. Rush;J. Siskind

Object classification from randomized EEG trials