III: EAGER: Learning Evaluation Metrics for Information Retrieval

点击翻译按钮获取中文摘要

Information retrieval (IR) performance is typically measured in terms of relevancy: every document is known to be either relevant or non-relevant to a particular query. Furthermore, more relevant documents are expected to receive a higher rank than lower less relevant documents. However, determination of relevance and rank by users is not practical. Therefore, it is crucial to develop evaluation metrics and ranking functions that can be derived automatically from judgment data and user behavior data, rather than ad-hoc heuristics. This exploratory project investigates machine learning approaches for constructing evaluation metrics for Web search and information retrieval that consider along important directions other than relevance such as diversity, balance and coverage. The approach is based on fundamentally extending the popular evaluation metric Discounted Cumulated Gains (DCG). Research focuses on developing optimization methods for learning DCG that can incorporate the degree of difference in pair-wise comparison of ranking lists. Machine learning methods that can learn DCG for the more realistic scenarios where the relevance grades are not readily available are explored, and nonlinear utility functions as evaluation metrics that can accurately capture the quality of search result sets in terms of relevance, diversity, coverage, balance and novelty are investigated.The project has a number of broad impacts. Research results are expected to provide foundations for further research in evaluation metrics. Active collaborations with industry leaders in Web search will enable the resulting methods to have real impacts on search engine as well as large IR system performance improvements. Improving the quality of search results will have significant impacts on satisfying people's information needs as well as their quality of life in general. The set of research topics lies at the interface between information retrieval and machine learning applications and it provides an ideal setting for training undergraduate and graduate students in the emerging interdisciplinary field of Web of science and engineering research. The project Web site (http://www.cc.gatech.edu/~zha/metrics.html) will be used for results dissemination.