A Study on Term-Document Clustering based on a co-evolutionary framework

在本研究中，我们提出了一种新的信息检索框架，我们称之为“基于聚类的索引”，并使用实际的文献集合来评估其有效性。该方案使用先前提出的“概率加权信息量”作为导航标准，在文档和术语之间同时进行聚类。其特点是，它旨在通过将术语和文档之间的关联视为传统检索系统中的“索引”来开发和利用它们。此外，该方法可以看作是对文本检索领域遗传算法中的“协同进化框架”的改进，因为它首先随机启动相邻术语和文档的聚类，然后对生成的聚类进行局部优化，以处理大规模的现实世界文档集合。在我们的研究中，我们还使用10,000 - 100,000个文档的测试集合来调查所提出方法的有效性；摘自NTCIR1的学术会议论文摘要，取自《每日新闻》和《日经新闻》光盘数据库的报纸文章，取自《路透社》或《金融时报》的英文报道。在使用文本分类任务的评估中，证实了生成的聚类的分类性能略差，但几乎与支持向量机的分类性能相当，支持向量机被认为是文本分类的最佳分类器之一。此外，该方法可以成功地提取类边界上文档之间的关联，这是传统的基于机器学习的分类方法所难以做到的。

In this study, we proposed a new framework of information retrieval, which we call "cluster-based indexing" , and evaluated the effectiveness using actual document collections.The proposed scheme employs simultaneous clustering between documents and terms using the previously proposed "probability weighted amount of information" as a navigation criteria. The feature is that it aims at exploiting and utilizing the extracted associations between terms and documents by treating them as 'indices' in conventional retrieval systems. Also, the proposed scheme can be considered as an adaptation of a "co-evolutionary framework" in genetic algorithms in the domain of text retrieval since it first randomly initiates clusters of neighboring terms and documents, and then, applies local optimization to the generated clusters in order to deal the large scale of real-world document collections.In our study, we also investigated the effectiveness of the proposed method using such test collections with 10,000 - 100,000 documents as ; abstracts of academic conference papers extracted from NTCIR1, newspaper articles from Mainichi and Nikkei CD-ROM databases, English stories from Reuters or Financial Times. In the evaluation using a text categorization task, it was confirmed that the categorization performance of the generated clusters was slightly worse but almost comparable to the one of Support Vector Machine, which is known to be one of the best classifier for text categorization. Furthermore, it was shown the method could successfully extract associations between documents on the class border, which is difficult with conventional machine-learning based categorization methods.