CAREER: Visual Recognition with Knowledge

职业：具有知识的视觉识别

• 批准号: 1750082
• 负责人: Yezhou Yang
• 金额: $ 55万
• 依托单位: Arizona State University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-08-15 至 2024-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1750082&HistoricalAwards=false
• 关键词: CAREER; Visual; Recognition; Knowledge

This project will address the problem of Visual Recognition with Knowledge (VR-K): a challenging Artificial Intelligence task to enable a seeing machine to identify unknown visible concepts from previous encounters (annotated data samples) and knowledge (other contextual information). For example, consider such a system that has never encountered a zebra, but which has previous visual encounters with "horses" and "black and white striped" patterns. Incorporating the linguistic input that, "A zebra is a horse-like animal with a black and white striped appearance", the machine's task is to formulate a new recognizer for the visual concept "zebra" and to recognize this new concept later. A system that integrates visual and linguistic information in this way can provide the basis for robust personal mobile applications or service robots, such as visual assistants to the vision-impaired, and voice-enable agents for elder care. Conventional supervised learning techniques have been perfected to perform increasingly well on narrow performance tasks. To enable satisfactory performance in service robots and mobile multimedia applications, this research will integrate background and commonsense knowledge models to enable higher level reasoning together with such high-performance recognizers. This project will develop the VR-K framework focused on enabling more generalizable computer vision algorithms through integration with natural language understanding and grounding in knowledge-based reasoning. The research program will include 1) developing efficient probabilistic reasoning engines to construct recognition models of unseen concepts (object and attribute) without new annotation through probabilistic semantic parsing; 2) setting up new large-scale visual challenges and testbeds as the basis for rigorous performance evaluation of visual recognition with knowledge models and ablation analysis; and 3) prototyping the proposed framework on service robots and mobile devices for evaluation of the proposed framework's performance in complex real-world applications over a variety of user studies. The project will include education and outreach activities advancing AI in undergraduate research, diversity enhancement, Entrepreneurial Mindset (EM) education, and K-12 classrooms, and will include workshops to introduce AI and deep learning to professionals in non-CS professions such as medical research and pathology.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该项目将解决带有知识的视觉识别（VR-K）问题：这是一项具有挑战性的人工智能任务，使视觉机器能够从以前的遭遇（带注释的数据样本）和知识（其他上下文信息）中识别未知的可见概念。例如，考虑这样一个系统，它从来没有遇到过斑马，但它以前在视觉上遇到过“马”和“黑白条纹”图案。结合语言输入，“斑马是一种像马一样的动物，有黑白条纹的外观”，机器的任务是为视觉概念“斑马”制定一个新的识别器，并在稍后识别这个新概念。以这种方式集成视觉和语言信息的系统可以为强大的个人移动应用程序或服务机器人提供基础，例如视障人士的视觉助手，以及老年人护理的语音代理。传统的监督学习技术已经得到完善，可以在狭义的任务中表现得越来越好。为了在服务机器人和移动多媒体应用中实现令人满意的性能，本研究将整合背景和常识知识模型，以实现更高层次的推理以及这些高性能识别器。该项目将开发VR-K框架，重点是通过集成自然语言理解和基于知识的推理，实现更通用的计算机视觉算法。研究项目将包括：1)开发高效的概率推理引擎，通过概率语义解析构建未见概念（对象和属性）的识别模型，无需新的注释；2)建立新的大规模视觉挑战和测试平台，以知识模型和消融分析作为严格的视觉识别性能评估的基础；3)在服务机器人和移动设备上对所提出的框架进行原型设计，以评估所提出的框架在各种用户研究的复杂实际应用中的性能。该项目将包括在本科研究、多样性增强、创业思维（EM）教育和K-12教室中推进人工智能的教育和推广活动，并将包括向医学研究和病理学等非计算机科学专业的专业人士介绍人工智能和深度学习的研讨会。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Modularized Textual Grounding for Counterfactual Resilience

• DOI: 10.1109/cvpr.2019.00654
• 发表时间: 2019-04
• 期刊: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: Zhiyuan Fang;Shu Kong;Charless C. Fowlkes;Yezhou Yang

Integrating Knowledge and Reasoning in Image Understanding

• DOI: 10.24963/ijcai.2019/873
• 发表时间: 2019-06
• 期刊: ArXiv
• 作者: Somak Aditya;Yezhou Yang;Chitta Baral

GAPLE: Generalizable Approaching Policy LEarning for Robotic Object Searching in Indoor Environment

• DOI: 10.1109/lra.2019.2930426
• 发表时间: 2018-09
• 期刊: IEEE Robotics and Automation Letters
• 影响因子: 5.2
• 作者: Xin Ye;Zhe L. Lin;Joon-Young Lee;Jianming Zhang;Shibin Zheng;Yezhou Yang

A Novel Design of Adaptive and Hierarchical Convolutional Neural Networks using Partial Reconfiguration on FPGA

• DOI: 10.1109/hpec.2019.8916237
• 发表时间: 2019-09
• 期刊: 2019 IEEE High Performance Extreme Computing Conference (HPEC)
• 作者: Mohammad Farhadi;Mehdi Ghasemi;Yezhou Yang

CAVAN: Commonsense Knowledge Anchored Video Captioning

• DOI: 10.1109/icpr56361.2022.9956241
• 发表时间: 2022-08
• 期刊: 2022 26th International Conference on Pattern Recognition (ICPR)
• 作者: Huiliang Shao;Zhiyuan Fang;Yezhou Yang