USING CORPUS STATISTICS TO REMOVE REDUNDANT WORDS IN TEXT CATEGORIZATION

利用语料库统计去除文本分类中的冗余单词

• 批准号: 2578635
• 负责人: W WILBUR
• 金额: --
• 依托单位: NATIONAL LIBRARY OF MEDICINE
• 依托单位国家: 美国
• 资助国家: 美国
• 起止时间: 至
• 项目状态: 未结题
• 来源: https://reporter.nih.gov/project-details/2578635
• 关键词: abstracting; artificial intelligence; information retrieval; statistics /biometry

In this collaborative project with Yiming Yang at Mayo Clinic, we use the term strength I have defined and use in the current Bayesian retrieval system for the Entrez neighbors, to determine thresholds for term removal. This allows a large number of terms to be identified as relatively useless. When these are removed the problem of text categorization based on the terms appearing in the text is greatly simplified. For the linear least squares fitting method developed and used by Dr. Yang, we find a time savings of 70 to 90% which comes from the removal of 80% or more of the terms. Dr. Yang has also developed what she terms an expert network method of text classification. It is based on finding the nearest neighbors to a text and using their classifications to predict the best classification for the text. The term removal methods provide significant time and space savings for this approach as well.

在这个与梅奥的杨一鸣合作的项目中