CAREER: Harnessing external knowledge to improve computer vision robustness, explainability, and user accuracy

该奖项的全部或部分资金来自《2021年美国救援计划法案》(公法117-2)。人工智能(AI)正在改变从交通到警察再到医疗保健的每一门学科。然而，基于人工智能的系统在面对前所未有的场景时经常出错。例如，自动驾驶汽车最关键的问题之一是无法处理边缘情况，导致在方向盘后面有人或没有人的情况下发生事故。这项研究的目标是建立基于人工智能的系统，利用外部知识来源(如维基百科)做出更知情的决策，从而做出更准确的决策。此外，人工智能系统有一个透明的决策过程，人类用户可以利用该过程做出准确的决策或调试人工智能系统。考虑到这些目标，该项目将解决关于制造基于人工智能的系统的三个研究问题。首先，该系统的设计应该是为了应对复杂、不断变化的现实世界中的新的、边缘的情况。其次，当人类是最终决策者时，人工智能系统应该最大限度地提高用户的准确性。最后，应该建立系统，以便人类用户可以调试和理解系统的决策过程。该项目的教育和外联活动包括在奥本大学开设一个新的可解释的人工智能课程，在K-6学校建立一个人工智能俱乐部，以及与业界合作。该项目解决了图像分类框架中的挑战，该框架利用外部知识库的显性视觉和文本知识来做出决策。为此，研究人员将利用大型文本语料库(例如，维基百科)和图像数据集作为外部信息来源，供图像分类器利用(例如，通过比较输入图像和支持图像)并做出更好的决策。研究目标是将图像分类器的范例从现有的仅依赖输入图像进行深度神经网络(DNN)的方法转变为使用DNN处理输入图像但也迭代地在图文数据库中搜索的混合半参数系统。这种范式转变将有助于提高基于人工智能的系统和人工智能团队决策的准确性。基于信息检索的计算机视觉方法将使基于人工智能的系统的决策过程：(A)对不适定的图像分类情况更稳健，例如在遮挡情况下；(B)用户本身更容易理解，因为人类可以自然地解释系统检索到的支持图像或文本；(C)它还可能导致图像分类任务的更稳健的DNN。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2).Artificial Intelligence (AI) is transforming every discipline from transportation to policing to healthcare. Yet, AI-based systems often make mistakes when facing scenarios that they have never seen before. For instance, one of the most critical issues with autonomous, self-driving cars is the inability to handle edge cases, causing accidents both with and without humans behind the steering wheel. The goals of this research are to build AI-based systems that harness external sources of knowledge (e.g., Wikipedia) to make more informed and thus more accurate decisions. In addition, the AI systems have a transparent decision-making process that human users can leverage to make accurate decisions or debug AI systems. With these goals in mind, the project will address three research questions on making AI-based systems. First, the system should be designed for addressing robust to new, edge cases in the complex, evolving real world. Second, the AI system should maximize user accuracy when humans are the end decision-makers. Finally, systems should be built so human users can debug and understand the systems’ decision-making process. Education and outreach activities of the project include a new Explainable AI course in Auburn University, an AI club in K-6 school, and collaboration with industry.The project addresses the challenges in image classification frameworks that harness explicit visual and textual knowledge from external knowledgebases to make decisions. Towards this, the researchers will utilize large text corpus (e.g., Wikipedia) and image datasets as an external source of information for image classifiers to leverage (for instance, by comparing the input image with support images) and make better decisions. The research goal is to shift the paradigm of image classifiers from the existing parametric approach that relies only on an input image for the deep neural networks (DNNs) into a hybrid, semi-parametric, system that uses DNNs to process the input image but also iteratively searches in an image-text database. This paradigm shift will help improve the accuracy of decision-making for both the AI-based systems and human-AI teams. The information-retrieval-based approach to computer vision will make the decision-making process of AI-based systems: (a) be more robust to ill-posed image classification cases e.g., under occlusions; (b) inherently more understandable to users as humans can naturally interpret the support images or text retrieved by the systems; (c) it could also lead to more robust DNNs for image classification tasks.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.