RI: Small: A Cognitive Framework for Technical, Hard and Explainable Question Answering (THE-QA) with respect to Combined Textual and Visual Inputs

对视觉和文本输入的理解是人工智能系统的重要方面。通常，这些输入被放在一起进行指导和解释。例如，智能机器人可以通过观察语言和手势来了解其任务和环境；解决科学问题的智能系统必须在解释文本的同时解释数字和图表。虽然有很多研究将视觉理解和文本理解分离开来，但将两者结合起来的研究却很少。该项目正在开发一个框架，用于回答有关视觉和文本组合输入的难题，并提供支持性解释。通过为这项任务开发一个集成视觉和语言信息的系统，该项目可以为K-12教育中的自动化辅导系统提供基础，并为操作智能机器的工人提供可解释的界面。该项目将采用基于深度模型的视觉识别和自然语言处理、知识表示和推理的综合方法来开发一个问答引擎及其组件。它将创建一个具有视觉和文本输入的挑战语料库，并以自然语言给出有关这些输入的问题。它将为语义图像和文本解析以及基于推理的问答系统提供一个基线。它将开发非连续文本项的语义解析，如图形、图表和图形。它将增强对各种格式的自然语言文本和问题的语义解析。它将发展方法来获取知识和推理与他们回答问题，并提供解释的答案。该项目的这些贡献将推动人工智能的发展，并使未来的服务机器人和个人移动应用程序能够理解视觉和文本输入的组合。该项目的研究结果将通过填补目前在如何有效地对深度模型进行可解释概率推理方面的空白，推动知识驱动、基于推理的问答的发展。这有助于克服训练好的视觉和文本理解模型的脆弱性。它还将通过探索解决难题的联合解决方案，揭示基于深度模型的视觉和语言理解算法与概率知识表示和推理之间的内在联系。总的来说，这个项目可能会在人工智能的多个子领域取得进展；即计算机视觉、自然语言处理和问答；并可能影响其他领域，比如机器人。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Understanding of visual and textual inputs are important aspects of Artificial Intelligence systems. Often such inputs are presented together to instruct and explain. As examples, an intelligent robot might learn about its tasks and environment by observing both language and gesture; and an intelligent system addressing scientific questions must interpret figures and diagrams along with text. While there has been a lot of research concerning visual understanding and textual understanding in isolation, there has been very little research that addresses them jointly. This project is developing a framework for answering hard questions about combined visual and textual inputs, and providing supporting explanations. By developing a system that integrates visual and linguistic information for this task, the project could provide the basis for automated tutoring systems in K-12 education, and interpretable interfaces for the workers operating intelligent machines. The project will employ an integrated approach of deep model-based visual recognition and natural language processing, and knowledge representation and reasoning to develop a question answering engine and its components. It will create a challenge corpus that has visual and textual inputs and questions about those inputs given in natural language. It will provide a baseline for semantic image and text parsing and reasoning-based question answering systems. It will develop semantic parsing of non-continuous text items, such as figures, diagrams, and graphs. It will enhance semantic parsing to various formats of natural language text and questions. It will develop methods to acquire knowledge and reasoning with them for answering questions and providing explanations to the answers. Together these contributions of the project will advance Artificial General Intelligence and allow future service robots and personal mobile applications to understand combined visual and textual inputs. The findings from this project will advance the development of knowledge-driven, reasoning-based question answering by filling the current gap on how to efficiently conduct explainable probabilistic reasoning over deep models. This helps to overcome the fragility of the trained visual and textual understanding models. It will also uncover the intrinsic connections between deep model-based vision and language understanding algorithms and probabilistic knowledge representation and reasoning by exploring a joint solution for answering the hard questions. In general, this project may result in advances in multiple sub-fields of Artificial Intelligence; namely, computer vision, natural language processing, and question answering; and may impact others such as robotics.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.