CAREER: Visual Recognition with Knowledge

职业：具有知识的视觉识别

• 批准号: 1750082
• 负责人: Yezhou Yang
• 金额: $ 55万
• 依托单位: Arizona State University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-08-15 至 2024-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1750082&HistoricalAwards=false
• 关键词: CAREER; Visual; Recognition; Knowledge

This project will address the problem of Visual Recognition with Knowledge (VR-K): a challenging Artificial Intelligence task to enable a seeing machine to identify unknown visible concepts from previous encounters (annotated data samples) and knowledge (other contextual information). For example, consider such a system that has never encountered a zebra, but which has previous visual encounters with "horses" and "black and white striped" patterns. Incorporating the linguistic input that, "A zebra is a horse-like animal with a black and white striped appearance", the machine's task is to formulate a new recognizer for the visual concept "zebra" and to recognize this new concept later. A system that integrates visual and linguistic information in this way can provide the basis for robust personal mobile applications or service robots, such as visual assistants to the vision-impaired, and voice-enable agents for elder care. Conventional supervised learning techniques have been perfected to perform increasingly well on narrow performance tasks. To enable satisfactory performance in service robots and mobile multimedia applications, this research will integrate background and commonsense knowledge models to enable higher level reasoning together with such high-performance recognizers. This project will develop the VR-K framework focused on enabling more generalizable computer vision algorithms through integration with natural language understanding and grounding in knowledge-based reasoning. The research program will include 1) developing efficient probabilistic reasoning engines to construct recognition models of unseen concepts (object and attribute) without new annotation through probabilistic semantic parsing; 2) setting up new large-scale visual challenges and testbeds as the basis for rigorous performance evaluation of visual recognition with knowledge models and ablation analysis; and 3) prototyping the proposed framework on service robots and mobile devices for evaluation of the proposed framework's performance in complex real-world applications over a variety of user studies. The project will include education and outreach activities advancing AI in undergraduate research, diversity enhancement, Entrepreneurial Mindset (EM) education, and K-12 classrooms, and will include workshops to introduce AI and deep learning to professionals in non-CS professions such as medical research and pathology.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该项目将解决视觉识别与知识（VR-K）的问题：一个具有挑战性的人工智能任务，使机器能够从以前的遭遇（注释数据样本）和知识（其他上下文信息）中识别未知的可见概念。例如，考虑这样的系统，其从未遇到斑马，但是其先前视觉上遇到“马”和“黑色和白色条纹”图案。该机器的任务是对视觉概念“斑马”进行新的识别，并在以后识别这个新的概念。以这种方式整合视觉和语言信息的系统可以为强大的个人移动的应用或服务机器人提供基础，例如视力受损者的视觉助手和老年护理的语音代理。传统的监督学习技术已经得到了完善，在狭窄的性能任务上表现得越来越好。为了使服务机器人和移动的多媒体应用程序的性能令人满意，本研究将整合背景和常识知识模型，使更高层次的推理与这种高性能的识别器。该项目将开发VR-K框架，重点是通过与自然语言理解的集成和基于知识的推理基础，实现更通用的计算机视觉算法。该研究计划将包括1）开发有效的概率推理引擎，以构建未知概念的识别模型（2）建立新的大规模视觉挑战和测试床，作为知识模型和消融分析对视觉识别进行严格性能评估的基础;以及3）在服务机器人和移动的设备上对所提出的框架进行原型设计，以在各种用户研究中评估所提出的框架在复杂的现实世界应用中的性能。该项目将包括教育和推广活动，推动人工智能在本科生研究，多样性增强，创业Minnesota（EM）教育和K-12教室，并将包括研讨会，向非专业人士介绍人工智能和深度学习。该奖项反映了NSF的法定使命，并通过使用基金会的知识产权进行评估，被认为值得支持。优点和更广泛的影响审查标准。

项目成果

Modularized Textual Grounding for Counterfactual Resilience

• DOI: 10.1109/cvpr.2019.00654
• 发表时间: 2019-04
• 期刊: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: Zhiyuan Fang;Shu Kong;Charless C. Fowlkes;Yezhou Yang

Integrating Knowledge and Reasoning in Image Understanding

• DOI: 10.24963/ijcai.2019/873
• 发表时间: 2019-06
• 期刊: ArXiv
• 作者: Somak Aditya;Yezhou Yang;Chitta Baral

GAPLE: Generalizable Approaching Policy LEarning for Robotic Object Searching in Indoor Environment

• DOI: 10.1109/lra.2019.2930426
• 发表时间: 2018-09
• 期刊: IEEE Robotics and Automation Letters
• 影响因子: 5.2
• 作者: Xin Ye;Zhe L. Lin;Joon-Young Lee;Jianming Zhang;Shibin Zheng;Yezhou Yang

A Novel Design of Adaptive and Hierarchical Convolutional Neural Networks using Partial Reconfiguration on FPGA

• DOI: 10.1109/hpec.2019.8916237
• 发表时间: 2019-09
• 期刊: 2019 IEEE High Performance Extreme Computing Conference (HPEC)
• 作者: Mohammad Farhadi;Mehdi Ghasemi;Yezhou Yang

CAVAN: Commonsense Knowledge Anchored Video Captioning

• DOI: 10.1109/icpr56361.2022.9956241
• 发表时间: 2022-08
• 期刊: 2022 26th International Conference on Pattern Recognition (ICPR)
• 作者: Huiliang Shao;Zhiyuan Fang;Yezhou Yang