RI: Small: Learning to Read, Ground, and Reason in Multimodal Text

Web数据、新闻和教科书提供信息丰富但非结构化的多模态文本。将多模态文本翻译成可进一步推理的语义表示的能力是驯服信息过载的关键一步，信息过载是现代人工智能的基本问题之一。设计能够理解和使用多模态文本的系统需要多个相互关联的组件：语义解释、多模态对齐、知识获取和推理。大多数先前的工作都集中在孤立的单个组件上，而忽略了这些任务之间的高阶关键相互依赖性。该提案旨在建立一个统一的框架&amp；#8232；努力学习阅读、理解和推理多模态教科书。它;# 8232;框架将包括三个相互连接的&amp；组件：上下文感知的视觉和文本解释，获取和表示知识，以及推理。这项工作旨在通过广泛的应用，包括教育和可访问性，产生重大的社会影响。在理解教科书和问题回答方面的进步可能有助于设计一个自动个性化的辅导系统，以教育学生代数、几何和科学主题。视觉口译和多模态知识的进步可以使视障人士更容易获得图表信息。该项目将有助于本科生和研究生的教育、研究和合作经验，包括代表性不足和少数群体。该框架旨在迭代地阅读上下文中的多模态教科书，获取知识，解释数据，更新和修剪所获取的知识，最后对查询进行推理。一个核心挑战是对多模态文本进行健壮的、可扩展的、上下文感知的语义分析和推理。该提案由三个主要推力组成，它们相互建立，以实现完整的提议框架。首先，该项目提出了一种精确的推理算法，在学习中解决代数单词问题。所提出的算法将学习使用叙事的全局上下文将局部上下文线索组合成一个新的语义结构。其次，提出了一个多模态文本的自动解释和推理系统，通过学习将文本和图表转化为形式表示，并提出了一种新的推理算法来解决这些问题。最后，它将构建一个新颖的、有原则的机器学习框架，用于多模态文本（科学教科书）中的知识获取、解释和推理。建议的框架将应用于会话对话和个性化辅导系统。主要贡献将包括一个统一的框架，用于学习阅读、理解和推理多模态教科书，用于联合多模态文本和图表解释的新算法，对叙述的精确理解，渐进的知识获取和推理。

Web data, news, and textbooks offer informative but unstructured multimodal text. The ability to translate multimodal text into a semantic representation that is amenable to further reasoning is a key step toward taming information overload, one of the fundamental problems in modern AI. Designing systems that can understand and use multimodal text requires multiple interconnected components: semantic interpretation, multimodal alignment, knowledge acquisition, and reasoning. Most previous work has focused on a single component in isolation and ignored the high-order crucial interdependencies between these tasks. This proposal aims at building a unified frame
work for learning to read, ground, and reason in multimodal textbooks. This
 framework will include three interconnected
 components: context-aware visual and textual interpretation, acquiring and representing knowledge, and reasoning. This work is designed for significant social impact through a broad range of applications including educational and accessibility. The advances in understanding textbooks and question answering could be potentially helpful in designing an automatic personalized tutoring system to educate students about algebra, geometry, and science topics. Advancements in visual interpretation and multimodal knowledge could be beneficial to visually impaired individuals to make the diagrammatic information accessible to them. This project will be instrumental for education, research, and collaborative experience for undergraduate and graduate students including under-represented and minority groups.The proposed framework is designed to iteratively read multimodal textbooks in context, acquire knowledge, interpret data, update and prune the acquired knowledge, and finally reason about the queries. A core challenge is to do robust, scalable, context-aware semantic analysis and reasoning on multimodal text. The proposal is organized in three main thrusts that build upon each other toward the complete proposed framework. First, the project proposes a precise reasoning algorithm in narratives in learning to solve algebra word problems. The proposed algorithm will learn to combine local contextual cues into a novel semantic structure using the global context of the narrative. Second, it proposes to build an automated system for interpreting and reasoning in multimodal text by learning to ground text and diagram into a formal representation and a new reasoning algorithm to solve those problems. Finally, it will construct a novel, principled machine learning framework for knowledge acquisition, interpretation, and reasoning in multimodal texts - science textbooks. The proposed framework will be applied in conversational dialogs and personalized tutoring systems. The key contributions will include a unified framework for learning to read, ground, and reason in multimodal textbooks, new algorithms for joint multi-modal text and diagram interpretation, precise understanding of narratives, gradual knowledge acquisition, and reasoning.