Automatic Identification of Significant Topics in Domain Independent Full Text Analysis

这项研究的目的是调查文档中重要主题的出现与文档结构之间的关系。这项研究的独特贡献在于将用于编制重要专题清单的方法结合起来，建立在统计和基于规则的技术基础上，以确定术语变体作为其在文件重点领域的分布的函数。可以使用这些方法的应用包括信息检索、段落检索、相关反馈、信息提取和摘要。这些成果可直接用于正在进行的文件自动摘要研究项目，使用统计和信息提取技术，即结合信息检索和自然语言处理。在某种程度上，这些技术是基于语言驱动的模式，而不是依赖于领域的词汇表，这些模式应该适用于一般文本。该方法将应用于多个领域，以测试其跨文档类型的通用性和适用性。这将允许衡量跨流派移植的成本。将在分析的每个步骤中制定和执行形成性和总结性评价程序。这项研究是在哥伦比亚大学数字图书馆研究计划的背景下，与信息获取研究中心一起进行的。由此产生的技术建立在信息检索和自然语言处理方法的新颖组合和相互促进的基础上，可望改善基于跨领域和流派的重要主题的信息获取。

The goal of this research is to investigate the relationship between the occurrence of significant topics in a document and the structure of the document. The unique contribution of this research lies in the combination of methods to be used for the production of a list of significant topics, built on both statistical and rule-based techniques for the identification of term variants as a function of their distribution in focus areas in documents. Applications that can employ these methods include information retrieval, passage retrieval, relevance feedback, information extraction, and summarization. The results can be used directly in ongoing research projects on automatic summarization of documents, using both statistical and information extraction techniques, i.e., combining information retrieval (IR) and natural language processing (NLP). To the extent that these techniques are based on linguistically-motivated patterns and not on domain-dependent vocabularies, these patterns should apply to general text. This approach will be applied to several domains to test its generality and applicability across document types. This will permit measuring the cost of porting across genres. Formative and summative evaluation procedures will be developed and performed at each step of the analysis. This research is undertaken in the context of the Digital Library Research Program at Columbia University, in conjunction with the Center for Research on Information Access. The resulting techniques grounded in the novel combination and cross-fertilization of IR and NLP methods are expected to improve information access based on significant topics across domains and genres.