RI: Small: Learning to Read, Ground, and Reason in Multimodal Text

网络数据、新闻和教科书提供了信息量大但无结构的多模式文本。将多模式文本转换为可进一步推理的语义表示的能力是驯服信息过载的关键一步，这是现代人工智能的基本问题之一。设计能够理解和使用多模式文本的系统需要多个相互关联的组件：语义解释、多模式对齐、知识获取和推理。以前的大多数工作都孤立地关注单个组件，而忽略了这些任务之间的高阶关键相互依赖关系。这项提议旨在建立一个统一的框架，学习多模式教科书中的阅读、基础和推理。这一框架将包括三个相互关联的组件：上下文感知的视觉和文本解释、获取和表示知识以及推理。这项工作旨在通过广泛的应用，包括教育和无障碍，产生重大的社会影响。在理解教科书和问题回答方面的进步可能有助于设计一个自动化的个性化辅导系统，以教育学生关于代数、几何和科学主题的知识。视觉解释和多模式知识的进步可能有利于视障人士使他们能够获得图解信息。该项目将有助于本科生和研究生的教育、研究和协作体验，包括代表不足和少数群体的学生。该框架旨在反复阅读上下文中的多模式教科书，获取知识，解释数据，更新和修剪已获得的知识，并最终对查询进行推理。一个核心挑战是对多模式文本进行健壮的、可伸缩的、上下文感知的语义分析和推理。该提案由三个主要部分组成，它们相互促进，形成了完整的提案框架。首先，该项目提出了一种在学习解决代数应用题的叙事中的精确推理算法。所提出的算法将学习使用叙事的全局上下文将局部上下文线索结合到新的语义结构中。其次，提出通过学习如何将文本和图表转化为形式表示和新的推理算法来解决这些问题，从而建立一个多模式文本的自动解释和推理系统。最后，它将构建一个新颖的、原则性的机器学习框架，用于多模式文本-科学教科书中的知识获取、解释和推理。所提出的框架将应用于对话对话和个性化辅导系统。主要贡献将包括在多模式教科书中学习阅读、基础和推理的统一框架，联合多模式文本和图表解释的新算法，对叙事的精确理解，逐步的知识获取和推理。

Web data, news, and textbooks offer informative but unstructured multimodal text. The ability to translate multimodal text into a semantic representation that is amenable to further reasoning is a key step toward taming information overload, one of the fundamental problems in modern AI. Designing systems that can understand and use multimodal text requires multiple interconnected components: semantic interpretation, multimodal alignment, knowledge acquisition, and reasoning. Most previous work has focused on a single component in isolation and ignored the high-order crucial interdependencies between these tasks. This proposal aims at building a unified frame
work for learning to read, ground, and reason in multimodal textbooks. This
 framework will include three interconnected
 components: context-aware visual and textual interpretation, acquiring and representing knowledge, and reasoning. This work is designed for significant social impact through a broad range of applications including educational and accessibility. The advances in understanding textbooks and question answering could be potentially helpful in designing an automatic personalized tutoring system to educate students about algebra, geometry, and science topics. Advancements in visual interpretation and multimodal knowledge could be beneficial to visually impaired individuals to make the diagrammatic information accessible to them. This project will be instrumental for education, research, and collaborative experience for undergraduate and graduate students including under-represented and minority groups.The proposed framework is designed to iteratively read multimodal textbooks in context, acquire knowledge, interpret data, update and prune the acquired knowledge, and finally reason about the queries. A core challenge is to do robust, scalable, context-aware semantic analysis and reasoning on multimodal text. The proposal is organized in three main thrusts that build upon each other toward the complete proposed framework. First, the project proposes a precise reasoning algorithm in narratives in learning to solve algebra word problems. The proposed algorithm will learn to combine local contextual cues into a novel semantic structure using the global context of the narrative. Second, it proposes to build an automated system for interpreting and reasoning in multimodal text by learning to ground text and diagram into a formal representation and a new reasoning algorithm to solve those problems. Finally, it will construct a novel, principled machine learning framework for knowledge acquisition, interpretation, and reasoning in multimodal texts - science textbooks. The proposed framework will be applied in conversational dialogs and personalized tutoring systems. The key contributions will include a unified framework for learning to read, ground, and reason in multimodal textbooks, new algorithms for joint multi-modal text and diagram interpretation, precise understanding of narratives, gradual knowledge acquisition, and reasoning.