CAREER: Harnessing external knowledge to improve computer vision robustness, explainability, and user accuracy

该奖项全部或部分由2021年美国救援计划法案（公法117-2）资助。人工智能（AI）正在改变从交通到警务到医疗保健的每一个学科。然而，基于人工智能的系统在面对从未见过的场景时经常会犯错误。例如，自动驾驶汽车最关键的问题之一是无法处理边缘情况，导致方向盘后面有人或无人的事故。这项研究的目标是建立基于人工智能的系统，利用外部知识来源（例如，维基百科），以作出更明智，从而更准确的决定。此外，人工智能系统有一个透明的决策过程，人类用户可以利用它来做出准确的决策或调试人工智能系统。考虑到这些目标，该项目将解决关于制作基于AI的系统的三个研究问题。首先，系统应该被设计为在复杂的、不断发展的真实的世界中解决新的、边缘的情况。其次，当人类是最终决策者时，人工智能系统应该最大限度地提高用户的准确性。最后，系统的构建应该使人类用户能够调试和理解系统的决策过程。该项目的教育和推广活动包括在奥本大学开设一门新的可解释人工智能课程，在K-6学校开设一个人工智能俱乐部，以及与行业合作。该项目解决了利用外部知识库中明确的视觉和文本知识进行决策的图像分类框架中的挑战。为此，研究人员将利用大型文本语料库（例如，维基百科）和图像数据集作为图像分类器的外部信息源，以利用（例如，通过将输入图像与支持图像进行比较）并做出更好的决策。研究目标是将图像分类器的范式从现有的仅依赖于深度神经网络（DNN）的输入图像的参数方法转变为混合的半参数系统，该系统使用DNN来处理输入图像，但也在图像-文本数据库中迭代搜索。这种范式转变将有助于提高基于AI的系统和人工智能团队的决策准确性。基于信息检索的计算机视觉方法将使基于AI的系统的决策过程：（a）对不适定的图像分类情况更加鲁棒，例如，在闭塞状态下;（B）由于人类可以自然地解释系统检索到的支持图像或文本，因此用户本质上更容易理解;（c）它还可以为图像分类任务带来更强大的DNN。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2).Artificial Intelligence (AI) is transforming every discipline from transportation to policing to healthcare. Yet, AI-based systems often make mistakes when facing scenarios that they have never seen before. For instance, one of the most critical issues with autonomous, self-driving cars is the inability to handle edge cases, causing accidents both with and without humans behind the steering wheel. The goals of this research are to build AI-based systems that harness external sources of knowledge (e.g., Wikipedia) to make more informed and thus more accurate decisions. In addition, the AI systems have a transparent decision-making process that human users can leverage to make accurate decisions or debug AI systems. With these goals in mind, the project will address three research questions on making AI-based systems. First, the system should be designed for addressing robust to new, edge cases in the complex, evolving real world. Second, the AI system should maximize user accuracy when humans are the end decision-makers. Finally, systems should be built so human users can debug and understand the systems’ decision-making process. Education and outreach activities of the project include a new Explainable AI course in Auburn University, an AI club in K-6 school, and collaboration with industry.The project addresses the challenges in image classification frameworks that harness explicit visual and textual knowledge from external knowledgebases to make decisions. Towards this, the researchers will utilize large text corpus (e.g., Wikipedia) and image datasets as an external source of information for image classifiers to leverage (for instance, by comparing the input image with support images) and make better decisions. The research goal is to shift the paradigm of image classifiers from the existing parametric approach that relies only on an input image for the deep neural networks (DNNs) into a hybrid, semi-parametric, system that uses DNNs to process the input image but also iteratively searches in an image-text database. This paradigm shift will help improve the accuracy of decision-making for both the AI-based systems and human-AI teams. The information-retrieval-based approach to computer vision will make the decision-making process of AI-based systems: (a) be more robust to ill-posed image classification cases e.g., under occlusions; (b) inherently more understandable to users as humans can naturally interpret the support images or text retrieved by the systems; (c) it could also lead to more robust DNNs for image classification tasks.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.