Automatic Identification of Significant Topics in Domain Independent Full Text Analysis

本研究的目的是探讨文献中重要主题的出现与文献结构之间的关系。本研究的独特贡献在于将用于生成重要主题列表的方法相结合，该列表基于统计和基于规则的技术，用于识别术语变体，作为它们在文档中重点领域分布的函数。可以使用这些方法的应用程序包括信息检索、通道检索、相关反馈、信息提取和摘要。这些结果可以直接用于正在进行的文档自动摘要研究项目，使用统计和信息提取技术，即结合信息检索（IR）和自然语言处理（NLP）。在某种程度上，这些技术是基于语言驱动的模式，而不是基于领域相关的词汇表，这些模式应该适用于一般文本。此方法将应用于多个领域，以测试其通用性和跨文档类型的适用性。这将有助于衡量跨类型移植的成本。形成性和总结性评估程序将在分析的每个步骤中制定和执行。这项研究是在哥伦比亚大学数字图书馆研究项目的背景下进行的，与信息获取研究中心合作。基于IR和NLP方法的新颖组合和交叉施肥的结果技术有望改善基于跨领域和体裁的重要主题的信息访问。

The goal of this research is to investigate the relationship between the occurrence of significant topics in a document and the structure of the document. The unique contribution of this research lies in the combination of methods to be used for the production of a list of significant topics, built on both statistical and rule-based techniques for the identification of term variants as a function of their distribution in focus areas in documents. Applications that can employ these methods include information retrieval, passage retrieval, relevance feedback, information extraction, and summarization. The results can be used directly in ongoing research projects on automatic summarization of documents, using both statistical and information extraction techniques, i.e., combining information retrieval (IR) and natural language processing (NLP). To the extent that these techniques are based on linguistically-motivated patterns and not on domain-dependent vocabularies, these patterns should apply to general text. This approach will be applied to several domains to test its generality and applicability across document types. This will permit measuring the cost of porting across genres. Formative and summative evaluation procedures will be developed and performed at each step of the analysis. This research is undertaken in the context of the Digital Library Research Program at Columbia University, in conjunction with the Center for Research on Information Access. The resulting techniques grounded in the novel combination and cross-fertilization of IR and NLP methods are expected to improve information access based on significant topics across domains and genres.