Data mining and knowledge discovery from unstructured text using cooperative and distributed algorithms

软件、硬件和网络技术的巨大进步使得海量数据的存储和共享成为可能。这些数据大多以非结构化文本的形式存在，无法使用传统的结构化数据库管理技术。在这项研究中，提出了管理非结构化数据的创新解决方案。我们专注于挖掘任务，如：根据文档的相似性将文档组织成类别(文档聚类)，以及使用针对大数据和高维数据的一组主题(文档分类)对文档进行语义标记。具体地说，我们提出了新的非结构化文本的有效表示，并使用分布式和协作式算法进行非结构化数据挖掘。在内容概念和语义的使用方面，建议的新表示法超越了传统方法。我们研究了使用代表性文档作为新语义空间的基础，在该空间中，文档之间的邻近度代表了它们的术语如何在统计上相关，从而提供了对文档之间真实语义相似性的更好估计。为了提高检索和分类任务的性能，提出了一种基于句子语义和统计分析的文档概念抽取方法。为了解决数据挖掘实际应用的大规模和分布式的特点，我们提出了一种新的分布式协作数据分类和聚类方法。这些方法利用在不同阶段合作的算法集合来改善最终输出。这些算法的自适应聚合和训练方面的创新想法有待开发。

The great advance in software, hardware, and network technologies allows the storage and sharing of a huge amount of data. Most of this data is in the form of unstructured text, where traditional techniques for managing structured databases could not be applied. In this research, innovative solutions to managing unstructured data are proposed. We focus on mining tasks such as: organization of documents into categories based on their similarity (document clustering) and semantic tagging of documents with a set of topics (document classification) for large and high dimensional data. In specific, we propose new efficient representations for unstructured text, and the use of distributive and collaborative algorithms for unstructured data mining. The proposed new representations go beyond the traditional approaches in terms of the use of content concepts and semantics. We investigate the use of representative documents as a basis for a new semantic space in which the proximity between documents represents how their terms are statistically correlated and accordingly provides a better estimate of the true semantic similarity between documents. Extraction of concepts from documents based on semantic and statistical analysis of sentences is proposed to enhance the performance of retrieval and categorization tasks. In order to address the large scale and distributed nature of real applications of data mining, we propose new distributive cooperative methods for data classification and clustering. These methods utilize ensemble of algorithms that cooperate at different stages to improve the final output. Innovative ideas in terms of adaptive aggregation and training of these algorithms are to be developed.