Constructing the Web-ware house for Web Mining

从国家资源的角度来看，WWW内容是日本非常重要的资源，因此这些内容有望对我们的重点行业有用和高效。然而，目前商业搜索引擎上的WWW内容仅用于关键字搜索，我们的研究目标是构建高性能的平台，提供对Web内容的可行访问，以研究新的WWW搜索方法。为此，我们研究了一种适合大数据密集型处理的系统体系结构、用于搜索海量数据的智能数据管理方法和用于高效利用Web仓库的日志分析方法。1)一种大规模的Web仓库系统体系结构：我们提出了通过存储区域网络(SAN)连接多个磁盘的PC集群系统，即SAN PC集群。对原型系统进行了实现，并对大型数据挖掘查询进行了评估。评估结果表明，该系统对于存储和处理WWW内容等大规模数据是有效的。2)智能处理海量WWW页面的数据管理方法：我们采用了一种基于WWW页面之间的超链接信息的新方法。这种方法与以前的方法完全不同。通过使用该方法，我们从收集并存储到我们的PC集群系统中的整个日语WWW页面中提取了社区图，并制作了WWW地图。3)WWW访问日志分析方法：对WWW访问日志进行分析，调查用户的行为，并将分析结果反映到电子商务网站、WWW商店等系统中。然后，从全局WWW日志(面板日志)中提取典型用户的行为模式。

WWW contents are very important resources in Japan from the point of country resources view, so these contents are expected to be useful and efficient for our key industries. However, the WWW contents are only utilized for keyword search on commercial search engines at this time.Our research goal is constructing high performance platform which provides the feasible access to Web contents in order to study novel search methods on WWW. So, we researched a novel system architecture which is appropriate for large data intensive processing, intelligent data management methods for searching large collected Web pages and log analyzing methods for providing an efficient utilization of Web-ware houses.1) A large scale system architecture for Web-ware houses : we proposed the PC cluster systems with many disks connected by SAN ( Storage Area Network), that is, SAN PC cluster. The prototype system is implemented and evaluated with large data mining query. The evaluation results show our proposed system is effective for storing and processing large scale data such as WWW contents.2) Data management methods for intelligent processing a large volume of WWW pages : we employed a novel approach based on hyper link information amongst WWW pages. This approach is completely different from previous methods. By using the proposed approach, we extracted a community chart from the whole Japanese WWW pages which are collected and stored into our PC cluster system and made a WWW map. Then, we provided a visualization tool supporting to search and show a various relationship among extracted communities.3) WWW access log analysis methods : we analyzed WWW access log to investigate user's behavior and to reflect analyzed results to systems such as e-commerce site, WWW store and so on. Then, we could extract typical user's behavior pattern from global WWW log ( panel logs ).