CAREER: Visual Recognition with Knowledge

该项目将解决知识视觉识别(VR-K)的问题：这是一项具有挑战性的人工智能任务，使视力机器能够从以前的遭遇中识别未知的可视概念(带注释的数据样本)和知识(其他上下文信息)。例如，考虑这样一个系统，它从未遇到过斑马，但之前在视觉上遇到了“马”和“黑白条纹”图案。这台机器的任务是为视觉概念“斑马”制定一个新的识别器，并在以后识别这个新的概念。以这种方式集成视觉和语言信息的系统可以为强大的个人移动应用程序或服务机器人提供基础，例如为视力受损的人提供视觉助手，以及为老年人护理提供语音支持的代理。传统的监督学习技术已经得到了完善，在狭隘的性能任务中表现得越来越好。为了在服务机器人和移动多媒体应用中实现令人满意的性能，本研究将整合背景和常识知识模型，以实现更高级别的推理和这样的高性能识别器。该项目将开发VR-K框架，重点是通过与自然语言理解和基于知识的推理相结合，实现更具普遍性的计算机视觉算法。该研究计划将包括：1)开发高效的概率推理引擎，通过概率语义分析，构建未见概念(对象和属性)的识别模型；2)建立新的大规模视觉挑战和试验台，作为使用知识模型和烧蚀分析进行严格视觉识别性能评估的基础；3)在服务机器人和移动设备上构建所提出的框架的原型，以评估所提出的框架在各种用户研究的复杂现实世界应用中的性能。该项目将包括在本科研究、增强多样性、企业家心态(EM)教育和K-12课堂中推进人工智能的教育和推广活动，并将包括向医学研究和病理学等非CS专业的专业人员介绍人工智能和深度学习的研讨会。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This project will address the problem of Visual Recognition with Knowledge (VR-K): a challenging Artificial Intelligence task to enable a seeing machine to identify unknown visible concepts from previous encounters (annotated data samples) and knowledge (other contextual information). For example, consider such a system that has never encountered a zebra, but which has previous visual encounters with "horses" and "black and white striped" patterns. Incorporating the linguistic input that, "A zebra is a horse-like animal with a black and white striped appearance", the machine's task is to formulate a new recognizer for the visual concept "zebra" and to recognize this new concept later. A system that integrates visual and linguistic information in this way can provide the basis for robust personal mobile applications or service robots, such as visual assistants to the vision-impaired, and voice-enable agents for elder care. Conventional supervised learning techniques have been perfected to perform increasingly well on narrow performance tasks. To enable satisfactory performance in service robots and mobile multimedia applications, this research will integrate background and commonsense knowledge models to enable higher level reasoning together with such high-performance recognizers. This project will develop the VR-K framework focused on enabling more generalizable computer vision algorithms through integration with natural language understanding and grounding in knowledge-based reasoning. The research program will include 1) developing efficient probabilistic reasoning engines to construct recognition models of unseen concepts (object and attribute) without new annotation through probabilistic semantic parsing; 2) setting up new large-scale visual challenges and testbeds as the basis for rigorous performance evaluation of visual recognition with knowledge models and ablation analysis; and 3) prototyping the proposed framework on service robots and mobile devices for evaluation of the proposed framework's performance in complex real-world applications over a variety of user studies. The project will include education and outreach activities advancing AI in undergraduate research, diversity enhancement, Entrepreneurial Mindset (EM) education, and K-12 classrooms, and will include workshops to introduce AI and deep learning to professionals in non-CS professions such as medical research and pathology.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.