Search engine using automatic web page ranking procedure and automatic classification

随着互联网的发展，万维网（WWW）页面的数量在过去几年中急剧增长。据估计，目前有超过32亿个万维网网页。为了满足新的搜索引擎对WWW网页的需求，需要开发出自动删除不重要网页、判断网页有用性和网页主题分类的机制，第一年开发了网页类型自动判断程序。网页被手动输入到标准页、首页、目录页、公告板、聊天页、链接页、日记页和输入表单。提出了一种基于判定页面定量分析的自动判定方法。从日文网页中获取HTML标签的出现频率、页面长度或标题和文件名中的单词，并根据这些信息进行类型判断。第二年，对网页的总量进行了估算，开发了有用网页的自动判断系统和自动分类系统。自动判断系统的算法是基于对网页的形态分析，通过判断“是良好的信息来源”获得高分。为了对WWW上的日文网页进行主题分类，本文提出了两种基于词的相对频率的分类算法和基于向量空间模型的信息检索技术，并将其应用于参加第二届NTCIR研讨会的Web任务的搜索引擎中。

The amount of World Wide Web (WWW) pages has grown dramatically over the last few years with the growth of internet. It is estimated that there are currently over 3,200 million WWW pages. In order to satisfy the requirement for new search engines for WWW pages, it is necessary to develop automatic mechanisms for the deletion of less important pages, judgment of usefulness of pages, and subject classification for Web pages.The first year, the automatic judging procedure for page type was developed. Web page were typed manually to standard pages, top pages, contents pages, bulletin boards, chat pages, link pages, diary pages, and input forms. The automatic judgment method based on quantitative analysis of judged pages was developed. The algorithm of a type judgment was based on the frequency of appearance of HTML tags, page length or words in titles and file names obtained from Web pages in Japanese.In the second year, the total amount of a Web page was estimated, and automatic judgment system of useful Web pages and automatic classification system were developed. The algorithm of automatic judgment system is based on the morphological analysis of pages which obtained the high score by the judgment of "being good sources of information". In order to classify WWW pages in Japanese by subject, we present two classification algorithms based on relative frequencies of terms and information retrieval technique using vector-space model.These methods are included in the search engine and it participated in the 2nd NTCIR workshop Web task.