Data mining and knowledge discovery from unstructured text using cooperative and distributed algorithms

点击翻译按钮获取中文摘要

The great advance in software, hardware, and network technologies allows the storage and sharing of a huge amount of data. Most of this data is in the form of unstructured text, where traditional techniques for managing structured databases could not be applied. In this research, innovative solutions to managing unstructured data are proposed. We focus on mining tasks such as: organization of documents into categories based on their similarity (document clustering) and semantic tagging of documents with a set of topics (document classification) for large and high dimensional data. In specific, we propose new efficient representations for unstructured text, and the use of distributive and collaborative algorithms for unstructured data mining. The proposed new representations go beyond the traditional approaches in terms of the use of content concepts and semantics. We investigate the use of representative documents as a basis for a new semantic space in which the proximity between documents represents how their terms are statistically correlated and accordingly provides a better estimate of the true semantic similarity between documents. Extraction of concepts from documents based on semantic and statistical analysis of sentences is proposed to enhance the performance of retrieval and categorization tasks. In order to address the large scale and distributed nature of real applications of data mining, we propose new distributive cooperative methods for data classification and clustering. These methods utilize ensemble of algorithms that cooperate at different stages to improve the final output. Innovative ideas in terms of adaptive aggregation and training of these algorithms are to be developed.