RI: Small: Learning to Read, Ground, and Reason in Multimodal Text

网络数据、新闻和教科书提供了信息丰富但非结构化的多模态文本。将多模态文本转换为可进一步推理的语义表示的能力是驯服信息过载的关键一步，信息过载是现代人工智能的基本问题之一。设计能够理解和使用多模态文本的系统需要多个相互关联的组件：语义解释，多模态对齐，知识获取和推理。大多数以前的工作集中在一个单独的组件，忽略了这些任务之间的高阶关键的相互依赖关系。本建议旨在建立一个统一的框架#8232;工作学习阅读，地面，并在多模态教科书的原因。 这个框架将包括三个相互关联的组件：上下文感知的视觉和文本解释，获取和表示知识，推理。这项工作旨在通过广泛的应用，包括教育和无障碍环境，产生重大的社会影响。理解教科书和问题回答方面的进步可能有助于设计自动个性化辅导系统，以教育学生代数，几何和科学主题。视觉解释和多模态知识的进步可能有利于视障人士，使他们能够获得图表信息。该项目将有助于教育，研究和合作经验的本科生和研究生，包括代表性不足和少数groups.The建议的框架的目的是迭代阅读多模态教科书的上下文中，获取知识，解释数据，更新和修剪所获得的知识，并最终的查询原因。 一个核心挑战是对多模态文本进行鲁棒的、可扩展的、上下文感知的语义分析和推理。该提案分为三个主要方面，它们相互依存，最终形成完整的拟议框架。首先，该项目提出了一个精确的推理算法，在学习中解决代数文字问题的叙述。 所提出的算法将学习联合收割机本地上下文线索结合成一个新的语义结构使用的全球背景下的叙事。其次，它提出了建立一个自动化系统的解释和推理多模态文本通过学习地面文本和图表到一个正式的表示和一个新的推理算法来解决这些问题。最后，它将构建一个新颖的，有原则的机器学习框架，用于多模态文本-科学教科书中的知识获取，解释和推理。该框架将应用于会话对话和个性化教学系统。主要贡献将包括一个统一的框架，用于学习阅读，地面和多模态教科书中的原因，联合多模态文本和图表解释的新算法，对叙述的精确理解，逐步知识获取和推理。

Web data, news, and textbooks offer informative but unstructured multimodal text. The ability to translate multimodal text into a semantic representation that is amenable to further reasoning is a key step toward taming information overload, one of the fundamental problems in modern AI. Designing systems that can understand and use multimodal text requires multiple interconnected components: semantic interpretation, multimodal alignment, knowledge acquisition, and reasoning. Most previous work has focused on a single component in isolation and ignored the high-order crucial interdependencies between these tasks. This proposal aims at building a unified frame
work for learning to read, ground, and reason in multimodal textbooks. This
 framework will include three interconnected
 components: context-aware visual and textual interpretation, acquiring and representing knowledge, and reasoning. This work is designed for significant social impact through a broad range of applications including educational and accessibility. The advances in understanding textbooks and question answering could be potentially helpful in designing an automatic personalized tutoring system to educate students about algebra, geometry, and science topics. Advancements in visual interpretation and multimodal knowledge could be beneficial to visually impaired individuals to make the diagrammatic information accessible to them. This project will be instrumental for education, research, and collaborative experience for undergraduate and graduate students including under-represented and minority groups.The proposed framework is designed to iteratively read multimodal textbooks in context, acquire knowledge, interpret data, update and prune the acquired knowledge, and finally reason about the queries. A core challenge is to do robust, scalable, context-aware semantic analysis and reasoning on multimodal text. The proposal is organized in three main thrusts that build upon each other toward the complete proposed framework. First, the project proposes a precise reasoning algorithm in narratives in learning to solve algebra word problems. The proposed algorithm will learn to combine local contextual cues into a novel semantic structure using the global context of the narrative. Second, it proposes to build an automated system for interpreting and reasoning in multimodal text by learning to ground text and diagram into a formal representation and a new reasoning algorithm to solve those problems. Finally, it will construct a novel, principled machine learning framework for knowledge acquisition, interpretation, and reasoning in multimodal texts - science textbooks. The proposed framework will be applied in conversational dialogs and personalized tutoring systems. The key contributions will include a unified framework for learning to read, ground, and reason in multimodal textbooks, new algorithms for joint multi-modal text and diagram interpretation, precise understanding of narratives, gradual knowledge acquisition, and reasoning.