RI: Small: A Cognitive Framework for Technical, Hard and Explainable Question Answering (THE-QA) with respect to Combined Textual and Visual Inputs

RI：小：结合文本和视觉输入的技术性、硬性和可解释性问答 (THE-QA) 的认知框架

• 批准号: 1816039
• 负责人: Chitta Baral
• 金额: $ 50万
• 依托单位: Arizona State University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-08-01 至 2024-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1816039&HistoricalAwards=false
• 关键词: RI; Small; Cognitive; Framework; Technical

Understanding of visual and textual inputs are important aspects of Artificial Intelligence systems. Often such inputs are presented together to instruct and explain. As examples, an intelligent robot might learn about its tasks and environment by observing both language and gesture; and an intelligent system addressing scientific questions must interpret figures and diagrams along with text. While there has been a lot of research concerning visual understanding and textual understanding in isolation, there has been very little research that addresses them jointly. This project is developing a framework for answering hard questions about combined visual and textual inputs, and providing supporting explanations. By developing a system that integrates visual and linguistic information for this task, the project could provide the basis for automated tutoring systems in K-12 education, and interpretable interfaces for the workers operating intelligent machines. The project will employ an integrated approach of deep model-based visual recognition and natural language processing, and knowledge representation and reasoning to develop a question answering engine and its components. It will create a challenge corpus that has visual and textual inputs and questions about those inputs given in natural language. It will provide a baseline for semantic image and text parsing and reasoning-based question answering systems. It will develop semantic parsing of non-continuous text items, such as figures, diagrams, and graphs. It will enhance semantic parsing to various formats of natural language text and questions. It will develop methods to acquire knowledge and reasoning with them for answering questions and providing explanations to the answers. Together these contributions of the project will advance Artificial General Intelligence and allow future service robots and personal mobile applications to understand combined visual and textual inputs. The findings from this project will advance the development of knowledge-driven, reasoning-based question answering by filling the current gap on how to efficiently conduct explainable probabilistic reasoning over deep models. This helps to overcome the fragility of the trained visual and textual understanding models. It will also uncover the intrinsic connections between deep model-based vision and language understanding algorithms and probabilistic knowledge representation and reasoning by exploring a joint solution for answering the hard questions. In general, this project may result in advances in multiple sub-fields of Artificial Intelligence; namely, computer vision, natural language processing, and question answering; and may impact others such as robotics.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

视觉和文本输入的理解是人工智能系统的重要方面。通常，这些输入一起呈现以指示和解释。 例如，智能机器人可以通过观察语言和手势来了解其任务和环境;解决科学问题的智能系统必须解释数字和图表沿着文本。虽然已经有很多关于视觉理解和文本理解的研究，但很少有研究将它们结合起来。该项目正在开发一个框架，用于回答有关结合视觉和文本输入的难题，并提供支持性解释。通过开发一个整合视觉和语言信息的系统，该项目可以为K-12教育中的自动辅导系统提供基础，并为操作智能机器的工人提供可解释的界面。该项目将采用基于深度模型的视觉识别和自然语言处理以及知识表示和推理的综合方法来开发问答引擎及其组件。它将创建一个挑战语料库，其中包含视觉和文本输入以及以自然语言给出的关于这些输入的问题。它将为语义图像和文本解析以及基于推理的问答系统提供基线。它将开发非连续文本项的语义分析，例如图形，图表和图表。它将增强对各种格式的自然语言文本和问题的语义解析。它将开发获取知识和推理的方法，以回答问题并为答案提供解释。该项目的这些贡献将共同推动人工通用智能，并允许未来的服务机器人和个人移动的应用程序理解组合的视觉和文本输入。该项目的研究结果将填补目前在如何有效地对深度模型进行可解释的概率推理方面的空白，从而推动知识驱动、基于推理的问答的发展。 这有助于克服训练的视觉和文本理解模型的脆弱性。它还将通过探索回答难题的联合解决方案，揭示基于模型的深度视觉和语言理解算法与概率知识表示和推理之间的内在联系。一般来说，该项目可能会在人工智能的多个子领域取得进展，即计算机视觉，自然语言处理和问答;并可能影响其他领域，如机器人技术。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

CLEVR_HYP: A Challenge Dataset and Baselines for Visual Question Answering with Hypothetical Actions over Images

• DOI: 10.18653/v1/2021.naacl-main.289
• 发表时间: 2021-04
• 期刊: ArXiv
• 作者: Shailaja Keyur Sampat;Akshay Kumar;Yezhou Yang;Chitta Baral

End-to-end Knowledge Retrieval with Multi-modal Queries

• DOI: 10.48550/arxiv.2306.00424
• 发表时间: 2023-06
• 作者: Man Luo;Zhiyuan Fang;Tejas Gokhale;Yezhou Yang;Chitta Baral

Video2Commonsense: Generating Commonsense Descriptions to Enrich Video Captioning

• DOI: 10.18653/v1/2020.emnlp-main.61
• 发表时间: 2020-03
• 作者: Zhiyuan Fang;Tejas Gokhale;Pratyay Banerjee;Chitta Baral;Yezhou Yang

Weakly Supervised Relative Spatial Reasoning for Visual Question Answering

• DOI: 10.1109/iccv48922.2021.00192
• 发表时间: 2021-09
• 期刊: 2021 IEEE/CVF International Conference on Computer Vision (ICCV)
• 作者: Pratyay Banerjee;Tejas Gokhale;Yezhou Yang;Chitta Baral

MUTANT: A Training Paradigm for Out-of-Distribution Generalization in Visual Question Answering

• DOI: 10.18653/v1/2020.emnlp-main.63
• 发表时间: 2020-09
• 作者: Tejas Gokhale;Pratyay Banerjee;Chitta Baral;Yezhou Yang