Study of High-speed Data Mining Algorithms from Massive Data Streams

本文研究了从海量半结构化数据流中提取有用信息的高速在线知识发现系统。特别是今年，在理论研究方面，我们进一步扩展了在线流的高效模式匹配和模式发现方法的理论。作为应用研究，我们对一个大型组织的真实高速网络进行了一系列的网络数据采集和分析实验。我们也发表了近三年的研究成果。特别是，我们在以下几个问题上进行了研究：(1)半结构化数据的调查：我们在学术期刊上总结并发表了一份关于流数据挖掘的调查报告，该报告通过本项目研究了三年。(2)半结构化数据流模式匹配技术研究：利用位并行技术开发了一种利用水平通配符执行树模式匹配的高效方法，为Xpath和XQuery模式匹配语言处理海量XML数据提供了极大的加速。(3)半结构化数据的顺序和流模式发现技术研究：开发了从海量数据流中发现各类复杂模式/图案的高效算法。今年，我们还发布了去年开发的模式发现算法。此外，其中一个还获得了2004年JSAI SIG奖。(4)基于真实海量网络数据的知识发现实证研究：作为应用，我们对九州大学中型组织进行了一系列关于高速大规模网络数据收集和在线分析的调查。这些实验将为未来研究高速流数据的高效模式匹配/发现算法的发展提供见解。

In this research, we investigated high-speed online knowledge discovery system for extracting useful information from massive semi-structured data streams. Particularly in this year, as theoretical researches, we extended further the theory of efficient pattern matching and pattern discovery methods for online streams. As application studies, we made a series of experiments on collection and analysis of network data from real high-speed networks in a huge organization. We have also published the results obtained in the research period of the last three years. In particular, we proceed the studies on the following issues:(1)Survey on semi-structured data : We have summarized and published a survey on stream data mining in an academic journal, which has been studied through this project for the last three years.(2)Study on streaming pattern matching technology for semi-structured data : We developed an efficient method for performing tree pattern matching with horizontal wildcards by bit parallel technology, which potentially gives drastic speed-up for Xpath and XQuery pattern matching languages for huge XML data.(3)Study on sequential and streaming pattern discovery technology for semi-structured data : We developed efficient algorithms for finding interesting patterns from massive data streams for various classes of complex patterns/motifs. In this year, we also published pattern discovery algorithms developed in the last year. Also, one of them got awarded for 2004 JSAI SIG AWARD.(4)Empirical study on knowledge discovery from real massive network data : As applications, we performed a series of surveys on data collection and online analysis of high-speed large-scale network for middle sized organization at Kyushu University. These experiments will give insights for future research on the development of efficient pattern matching/discovery algorithms for high-speed streaming data.