Search engine using automatic web page ranking procedure and automatic classification

在过去的几年里，随着互联网的发展，万维网（WWW）页面的数量急剧增长。据估计，目前有超过32亿个WWW页面。为了满足新的WWW页面搜索引擎的需求，有必要开发网页的自动删除机制、网页有用性判断机制和网页主题分类机制。第一年，开发了页码的自动判断程序。网页被手动输入到标准页、首页、内容页、公告栏、聊天页、链接页、日记页和输入表单。提出了基于页数定量分析的自动判断方法。类型判断算法基于从日语网页中获取的HTML标签出现频率、页面长度或标题和文件名中的单词。第二年，对网页总量进行估算，开发了有用网页自动判断系统和自动分类系统。自动判断系统的算法是基于对被判定为“好信息源”而获得高分的页面进行形态分析。为了按主题对日语WWW页面进行分类，提出了两种基于词条相对频率的分类算法和基于向量空间模型的信息检索技术。这些方法被纳入搜索引擎，并参加了第二届NTCIR研讨会的Web任务。

The amount of World Wide Web (WWW) pages has grown dramatically over the last few years with the growth of internet. It is estimated that there are currently over 3,200 million WWW pages. In order to satisfy the requirement for new search engines for WWW pages, it is necessary to develop automatic mechanisms for the deletion of less important pages, judgment of usefulness of pages, and subject classification for Web pages.The first year, the automatic judging procedure for page type was developed. Web page were typed manually to standard pages, top pages, contents pages, bulletin boards, chat pages, link pages, diary pages, and input forms. The automatic judgment method based on quantitative analysis of judged pages was developed. The algorithm of a type judgment was based on the frequency of appearance of HTML tags, page length or words in titles and file names obtained from Web pages in Japanese.In the second year, the total amount of a Web page was estimated, and automatic judgment system of useful Web pages and automatic classification system were developed. The algorithm of automatic judgment system is based on the morphological analysis of pages which obtained the high score by the judgment of "being good sources of information". In order to classify WWW pages in Japanese by subject, we present two classification algorithms based on relative frequencies of terms and information retrieval technique using vector-space model.These methods are included in the search engine and it participated in the 2nd NTCIR workshop Web task.