USING CORPUS STATISTICS TO REMOVE REDUNDANT WORDS IN TEXT CATEGORIZATION

在这个与梅奥诊所的杨一鸣合作的项目中，我们使用 我已经在当前的贝叶斯检索中定义并使用了术语强度 Entrez邻居的系统，以确定术语的阈值 移走。这允许大量的术语被标识为 相对无用。当这些被移除时，文本的问题 基于文本中出现的术语的分类在很大程度上 简化了。对于开发的线性最小二乘拟合法和 杨博士使用，我们发现节省了70%到90%的时间，这来自于 删除80%或更多的条款。杨博士还发现了 她所说的一种专家网络文本分类方法。它是 基于查找与文本最近的邻居并使用其 预测文本的最佳分类的分类。这个 术语删除方法可为此节省大量时间和空间 也可以进场。

In this collaborative project with Yiming Yang at Mayo Clinic, we use the term strength I have defined and use in the current Bayesian retrieval system for the Entrez neighbors, to determine thresholds for term removal. This allows a large number of terms to be identified as relatively useless. When these are removed the problem of text categorization based on the terms appearing in the text is greatly simplified. For the linear least squares fitting method developed and used by Dr. Yang, we find a time savings of 70 to 90% which comes from the removal of 80% or more of the terms. Dr. Yang has also developed what she terms an expert network method of text classification. It is based on finding the nearest neighbors to a text and using their classifications to predict the best classification for the text. The term removal methods provide significant time and space savings for this approach as well.