Data mining and knowledge discovery from unstructured text using cooperative and distributed algorithms

软件、硬件和网络技术的巨大进步使得海量数据得以存储和共享。大多数数据都是非结构化文本的形式，无法应用管理结构化数据库的传统技术。 在这项研究中，提出了管理非结构化数据的创新解决方案。我们专注于挖掘任务，例如：根据文档的相似性将文档组织为类别（文档聚类），以及针对大型和高维数据使用一组主题对文档进行语义标记（文档分类）。具体来说，我们提出了非结构化文本的新有效表示，以及使用分布式和协作算法进行非结构化数据挖掘。所提出的新表示在内容概念和语义的使用方面超越了传统方法。我们研究使用代表性文档作为新语义空间的基础，其中文档之间的接近度表示它们的术语在统计上如何相关，从而更好地估计文档之间的真实语义相似性。提出了基于句子语义和统计分析的文档概念提取，以提高检索和分类任务的性能。为了解决数据挖掘实际应用的大规模和分布式性质，我们提出了新的数据分类和聚类的分布式协作方法。 这些方法利用在不同阶段合作的算法集合来提高最终输出。将开发这些算法的自适应聚合和训练方面的创新想法。

The great advance in software, hardware, and network technologies allows the storage and sharing of a huge amount of data. Most of this data is in the form of unstructured text, where traditional techniques for managing structured databases could not be applied. In this research, innovative solutions to managing unstructured data are proposed. We focus on mining tasks such as: organization of documents into categories based on their similarity (document clustering) and semantic tagging of documents with a set of topics (document classification) for large and high dimensional data. In specific, we propose new efficient representations for unstructured text, and the use of distributive and collaborative algorithms for unstructured data mining. The proposed new representations go beyond the traditional approaches in terms of the use of content concepts and semantics. We investigate the use of representative documents as a basis for a new semantic space in which the proximity between documents represents how their terms are statistically correlated and accordingly provides a better estimate of the true semantic similarity between documents. Extraction of concepts from documents based on semantic and statistical analysis of sentences is proposed to enhance the performance of retrieval and categorization tasks. In order to address the large scale and distributed nature of real applications of data mining, we propose new distributive cooperative methods for data classification and clustering. These methods utilize ensemble of algorithms that cooperate at different stages to improve the final output. Innovative ideas in terms of adaptive aggregation and training of these algorithms are to be developed.