Data mining and knowledge discovery from unstructured text using cooperative and distributed algorithms

软件、硬件和网络技术的巨大进步允许存储和共享大量数据。这些数据中的大多数是非结构化文本，无法应用管理结构化数据库的传统技术。 在这项研究中，提出了管理非结构化数据的创新解决方案。我们专注于挖掘任务，如：组织的文件到类别的基础上，他们的相似性（文件聚类）和语义标记的文件与一组主题（文件分类）的大型和高维数据。具体而言，我们提出了新的非结构化文本的高效表示，并使用分布式和协作算法进行非结构化数据挖掘。所提出的新的表示超越了传统的方法，在使用的内容概念和语义。我们调查使用的代表性文件作为一个新的语义空间的基础上，文件之间的接近表示他们的条款是如何统计相关的，并因此提供了一个更好的估计文件之间的真实语义相似性。提出了一种基于句子语义和统计分析的概念抽取方法，以提高检索和分类任务的性能。为了解决数据挖掘真实的应用的大规模和分布式的性质，我们提出了新的分布式协作方法的数据分类和聚类。 这些方法利用在不同阶段合作的算法的集合来改善最终输出。在这些算法的自适应聚合和训练方面的创新想法有待开发。

The great advance in software, hardware, and network technologies allows the storage and sharing of a huge amount of data. Most of this data is in the form of unstructured text, where traditional techniques for managing structured databases could not be applied. In this research, innovative solutions to managing unstructured data are proposed. We focus on mining tasks such as: organization of documents into categories based on their similarity (document clustering) and semantic tagging of documents with a set of topics (document classification) for large and high dimensional data. In specific, we propose new efficient representations for unstructured text, and the use of distributive and collaborative algorithms for unstructured data mining. The proposed new representations go beyond the traditional approaches in terms of the use of content concepts and semantics. We investigate the use of representative documents as a basis for a new semantic space in which the proximity between documents represents how their terms are statistically correlated and accordingly provides a better estimate of the true semantic similarity between documents. Extraction of concepts from documents based on semantic and statistical analysis of sentences is proposed to enhance the performance of retrieval and categorization tasks. In order to address the large scale and distributed nature of real applications of data mining, we propose new distributive cooperative methods for data classification and clustering. These methods utilize ensemble of algorithms that cooperate at different stages to improve the final output. Innovative ideas in terms of adaptive aggregation and training of these algorithms are to be developed.