RI: Small: Modeling Multiple Modalities for Knowledge-Base Construction

RI：小型：知识库构建的多种模式建模

• 批准号: 1817183
• 负责人: Sameer Singh
• 金额: $ 44.8万
• 依托单位: University of California-Irvine
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-08-01 至 2022-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1817183&HistoricalAwards=false
• 关键词: RI; Small; Modeling; Multiple; Modalities

Information-rich documents are prevalent in many domains such as news articles, social media posts, online retail pages, healthcare records, financial reports, and scientific papers. Automatically extracting knowledge from such documents is useful in many applications, such as for answering questions, searching the web, automated dialogs, and analyzing trends. Existing machine learning methods focus only on the text in the documents, and ignore other information sources such as images, tables, and numbers. Thus, much of the information is not extracted, leading to incomplete knowledge and incorrect conclusions. This research advances research in machine learning and natural language processing to address these problems. With support for accurate extraction and reasoning, this project will pave the way for novel applications to domains with unstructured, multimodal documents.The specific aim of the project is to investigate a novel construction pipeline for knowledge bases, taking the first steps in combining textual and relational evidence with numerical, image, and tabular data. To address the many interconnected challenges therein, the project focuses on two sub-tasks. First, the team will extract new facts about an entity from a document, such as its attributes, by combining the different parts (text, images, and tables). Second, the team will develop models to identify missing relations in graphs that contain multimodal facts. For each task, the project includes plans to introduce new datasets, propose benchmark evaluations, and develop appropriate baselines. Further, the team will build upon recent advances in deep neural encoders to investigate machine learning approaches that learn unified, semantic embeddings to model multimodal data. With these contributions, the project will initiate a body of research in machine learning and natural language processing that uses unstructured multimodal data in all its forms for accurate knowledge extraction.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

信息丰富的文档普遍存在于许多领域，如新闻文章、社交媒体帖子、在线零售页面、医疗记录、财务报告和科学论文。从这样的文档中自动提取知识在许多应用程序中都很有用，例如用于回答问题、搜索Web、自动对话和分析趋势。现有的机器学习方法只关注文档中的文本，而忽略了图像、表格和数字等其他信息源。因此，许多信息没有被提取，导致知识不完整和不正确的结论。为了解决这些问题，本研究推进了机器学习和自然语言处理的研究。在对准确提取和推理的支持下，该项目将为非结构化、多模式文档领域的新型应用铺平道路。该项目的具体目标是研究一种新的知识库构建管道，迈出将文本和关系证据与数字、图像和表格数据相结合的第一步。为了应对其中许多相互关联的挑战，该项目侧重于两个子任务。首先，团队将通过组合不同的部分(文本、图像和表格)从文档中提取有关实体的新事实，如其属性。其次，该团队将开发模型，以识别包含多模式事实的图表中缺失的关系。对于每项任务，该项目包括引入新的数据集、提出基准评估和制定适当基线的计划。此外，该团队将以深度神经编码器的最新进展为基础，研究学习统一的语义嵌入来对多模式数据建模的机器学习方法。有了这些贡献，该项目将启动一系列机器学习和自然语言处理的研究，使用各种形式的非结构化多模式数据进行准确的知识提取。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Tweeki: Linking Named Entities on Twitter to a Knowledge Graph

• DOI: 10.18653/v1/2020.wnut-1.29
• 发表时间: 2020-11
• 期刊: ArXiv
• 作者: Bahareh Harandizadeh;Sameer Singh

Knowledge Enhanced Contextual Word Representations

• DOI: 10.18653/v1/d19-1005
• 发表时间: 2019-09
• 期刊: ArXiv
• 作者: Matthew E. Peters;Mark Neumann;IV RobertL.Logan;Roy Schwartz;Vidur Joshi;Sameer Singh;Noah A. Smith-

Embedding Multimodal Relational Data for Knowledge Base Completion

• DOI: 10.18653/v1/d18-1359
• 发表时间: 2018-09
• 期刊: ArXiv
• 作者: Pouya Pezeshkpour;Liyan Chen;Sameer Singh

Enforcing Consistency in Weakly Supervised Semantic Parsing

增强弱监督语义解析的一致性

• DOI: 10.18653/v1/2021.acl-short.22
• 发表时间: 2021
• 期刊: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers
• 作者: Gupta, Nitish;Singh, Sameer;Gardner, Matt
• 通讯作者: Gardner, Matt

Entity-Based Knowledge Conflicts in Question Answering

• DOI: 10.18653/v1/2021.emnlp-main.565
• 发表时间: 2021-09
• 期刊: ArXiv
• 作者: S. Longpre;Kartik Perisetla;Anthony Chen;Nikhil Ramesh;Chris DuBois;Sameer Singh