Research on Unified Discovery of Exceptions from Massive Data

本研究的目的是研究和开发一种基于各种学习方法的数据挖掘方法，以统一的方式从大量数据中发现有趣的异常，并通过真实数据集的实验来证明其有效性。2001财政年度的进展情况如下：(1)开发和改进了基于支持向量机、bloomy决策树、异常规则发现和boosting的各种异常发现方法。我们主要致力于数据压缩，以应对海量数据。(2)对所开发的异常发现方法进行了实验评价。我们还总结了数据挖掘竞赛，每一场竞赛都代表了对各种知识发现方法进行系统评估的场合。(3)规划和研究统一的异常发现方法。我们还执行了一种新的规则发现的最坏情况分析，作为自动发现的基础。在2002财政年度，我们首先开发了统一的异常规则发现，并在计算机上实现了它。集成中的重要问题包括发现知识的有用性和基于上一财年实验结果的方法的有效性。在开发过程中，每个异常发现方法都在必要时进行了改进。根据前期实验的结果，在上一财年开发完善的异常发现方法中，我们选择了采用异常规则发现方法和基于boosting的离群值检测方法的统一异常发现方法作为最终系统。在本财年下半年，我们进行了最后的实验，将实现的统一异常发现方法应用于预处理的海量数据。

The objective of this research is to study and develop a data mining method which discovers interesting exceptions from massive data in a uniform way based on various learning methods, and to justify its effectiveness by experiments with real data sets.The progress in fiscal year 2001 consists of the followings. (1) Development and refinement of various exception discovery methods including those based on support vector machines, bloomy decision tree, exception rule discovery, and boosting. We mainly worked on data squashing in order to cope with massive data. (2) Experimental evaluation of the developed exception discovery methods. We also summarized data mining contests each of which represents an occasion of systematic evaluation for various knowledge discovery methods with a set of common problems. (3) Planning and investigation of a unified exception discovery method. We also performed a novel type of worst-case analysis of rule discovery as a foundation of automated discovery.In fiscal year 2002, we first developed a unified exception rule discovery and implemented it on computers. Important issues in the integration include usefulness of discovered knowledge and effectiveness of the approach based on the experimental results in the previous fiscal year, In the development, each exception discovery method was refined if necessary. According to the results of preliminary experiments, we have chosen the unified exception discovery method which employs exception rule discovery method and outlier detection method based on boosting as our final system among the exception discovery methods developed and refined in the last fiscal year. In the latter half of this fiscal year, we performed final experiments in which we applied the implemented unified exception discovery method to preprocessed massive data.