Constructing the Web-ware house for Web Mining

从国家资源的角度来看，WWW内容在日本是非常重要的资源，因此这些内容有望对我们的重点产业有用和有效。然而，目前商业搜索引擎仅能利用WWW内容进行关键词搜索，我们的研究目标是构建一个高性能的平台，为Web内容的访问提供可行的途径，以研究新颖的WWW搜索方法。因此，我们研究了一种新的适合于大数据密集型处理的系统架构、用于搜索大规模收集的Web页面的智能数据管理方法和用于提供有效利用Web仓库的日志分析方法。1）一种适用于Web仓库的大规模系统架构：本文提出了一种多磁盘通过SAN（Storage Area Network）连接的PC集群系统，即SAN PC集群。原型系统的实现和评估与大型数据挖掘查询。评估结果表明，我们提出的系统是有效的存储和处理大规模的数据，如WWW内容。2）数据管理方法，智能处理大量的WWW页面：我们采用了一种新的方法，基于超链接信息之间的WWW页面。这种方法与以前的方法完全不同。通过使用所提出的方法，我们提取了一个社区图从整个日本的WWW网页收集和存储到我们的PC集群系统，并作出了WWW地图。3）WWW访问日志分析方法：通过对WWW访问日志的分析，研究用户的行为，并将分析结果反映到电子商务网站、WWW商店等系统中，然后从全局WWW日志（面板日志）中提取典型用户的行为模式。

WWW contents are very important resources in Japan from the point of country resources view, so these contents are expected to be useful and efficient for our key industries. However, the WWW contents are only utilized for keyword search on commercial search engines at this time.Our research goal is constructing high performance platform which provides the feasible access to Web contents in order to study novel search methods on WWW. So, we researched a novel system architecture which is appropriate for large data intensive processing, intelligent data management methods for searching large collected Web pages and log analyzing methods for providing an efficient utilization of Web-ware houses.1) A large scale system architecture for Web-ware houses : we proposed the PC cluster systems with many disks connected by SAN ( Storage Area Network), that is, SAN PC cluster. The prototype system is implemented and evaluated with large data mining query. The evaluation results show our proposed system is effective for storing and processing large scale data such as WWW contents.2) Data management methods for intelligent processing a large volume of WWW pages : we employed a novel approach based on hyper link information amongst WWW pages. This approach is completely different from previous methods. By using the proposed approach, we extracted a community chart from the whole Japanese WWW pages which are collected and stored into our PC cluster system and made a WWW map. Then, we provided a visualization tool supporting to search and show a various relationship among extracted communities.3) WWW access log analysis methods : we analyzed WWW access log to investigate user's behavior and to reflect analyzed results to systems such as e-commerce site, WWW store and so on. Then, we could extract typical user's behavior pattern from global WWW log ( panel logs ).