Domain-Independent Unsupervised Text Segmentation for Data Management
Domain-Independent Unsupervised Text Segmentation for Data Management
复制标题
用于数据管理的与领域无关的无监督文本分割
DOI：
10.1109/icdmw.2014.118
复制
发表时间：
2014
期刊：
影响因子：
--
通讯作者：
Katsumi Nitta
中科院分区：
文献类型：
--
作者：
Makoto Sakahara;Shogo Okada;Katsumi Nitta
在这项研究中，我们提出了一个领域无关的无监督文本分割方法，这是适用于即使看不见的单一文件。该方法通过评估句子之间的相似性来分割文本文档。当领域知识不足时，通常难以计算组成句子的词之间的语义相似度。这个问题影响了分割的准确性。为了解决这个问题，我们使用word2vec来计算单词之间的语义相似度。使用word 2 vec，我们通过使用大型领域无关语料库进行训练，将单词之间的语义关系嵌入到向量空间中。此外，我们结合联合收割机语义和搭配的相似性，即，文档中单词之间的特征。所提出的方法将这种组合相似性应用于亲和力传播聚类。句子之间的相似性是根据推土机的频率之间的距离所获得的主题集群。在计算句子之间的相似度之后，使用动态规划自动优化分割边界。使用两个数据集获得的实验结果表明，所提出的方法明显优于最先进的领域独立的方法，并获得与最先进的领域相关的方法，如使用主题建模相同的性能。
In this study, we have proposed a domain-independent unsupervised text segmentation method, which is applicable to even if unseen single document. This proposed method segments text documents by evaluating similarity between sentences. It is generally difficult to calculate semantic similarity between words that comprise sentences when the domain knowledge is insufficient. This problem influences segmentation accuracy. To address this problem, we use word 2 vec to calculate semantic similarity between words. Using word 2 vec, we embed semantic relationships between words in a vector space by training with large domain-independent corpora. Furthermore, we combine semantic and collocation similarities, i.e., The features between words within a document. The proposed method applies this combined similarity to affinity propagation clustering. Similarity between sentences is defined based on the earth mover's distance between the frequencies of the obtained topical clusters. After calculating similarity between sentences, segmentation boundaries are automatically optimized using dynamic programming. The experimental results obtained using two datasets show that the proposed method clearly outperforms state-of-the-art domain-independent approaches and obtains equal performance with state-of-the-art domain-dependent approaches such as those that use topic modeling.