USING CORPUS STATISTICS TO REMOVE REDUNDANT WORDS IN TEXT CATEGORIZATION

在这个与杨一鸣在马约的合作项目中， 在临床上，我们使用我定义和使用的术语强度， 目前的贝叶斯检索系统， 邻居，以确定删除术语的阈值。这 允许大量的术语被识别为 相对无用。当这些被移除时， 文本分类的基础上出现的条款 文本大大简化。对于线性最小二乘 杨博士开发和使用的拟合方法，我们发现一个 节省70%到90%的时间，这是因为 80%或以上的条款。杨博士还研究出了 她称之为文本分类的专家网络方法。 它基于查找文本的最近邻居， 用他们的分类来预测最好的 文本的分类。术语去除方法 节省大量时间和空间 方法也。

In this collaborative project with Yiming Yang at Mayo Clinic, we use the term strength I have defined and use in the current Bayesian retrieval system for the Entrez neighbors, to determine thresholds for term removal. This allows a large number of terms to be identified as relatively useless. When these are removed the problem of text categorization based on the terms appearing in the text is greatly simplified. For the linear least squares fitting method developed and used by Dr. Yang, we find a time savings of 70 to 90% which comes from the removal of 80% or more of the terms. Dr. Yang has also developed what she terms an expert network method of text classification. It is based on finding the nearest neighbors to a text and using their classifications to predict the best classification for the text. The term removal methods provide significant time and space savings for this approach as well.