III: EAGER: Learning Evaluation Metrics for Information Retrieval

信息检索（IR）性能通常是根据相关性来衡量的：每个文档都已知与特定查询相关或不相关。此外，更相关的文件预计将获得比较低的不太相关的文件更高的排名。然而，由用户确定相关性和等级是不实际的。因此，开发可以从判断数据和用户行为数据自动导出的评估指标和排名函数，而不是ad-hoc算法是至关重要的。这个探索性的项目研究机器学习方法，用于构建Web搜索和信息检索的评估指标，这些指标考虑沿着重要的方向，而不是相关性，如多样性，平衡性和覆盖率。该方法是基于从根本上扩展流行的评估指标贴现累积收益（DCG）。研究重点是开发用于学习DCG的优化方法，该方法可以将排序列表的成对比较中的差异程度结合起来。机器学习方法，可以学习DCG的更现实的情况下，相关性等级是不容易获得的探索，和非线性效用函数作为评估指标，可以准确地捕捉搜索结果集的相关性，多样性，覆盖率，平衡性和新奇方面的质量进行了调查。研究结果有望为评价指标的进一步研究提供基础。与Web搜索行业领导者的积极合作将使所产生的方法对搜索引擎产生真实的影响，并大大提高IR系统的性能。提高搜索结果的质量将对满足人们的信息需求以及他们的生活质量产生重大影响。该研究课题集位于信息检索和机器学习应用程序之间的接口，它提供了一个理想的环境，培养本科生和研究生在新兴的跨学科领域的Web科学和工程研究。将利用项目网址（http：//www.cc.gatech.edu/martzha/metrics.html）传播成果。

Information retrieval (IR) performance is typically measured in terms of relevancy: every document is known to be either relevant or non-relevant to a particular query. Furthermore, more relevant documents are expected to receive a higher rank than lower less relevant documents. However, determination of relevance and rank by users is not practical. Therefore, it is crucial to develop evaluation metrics and ranking functions that can be derived automatically from judgment data and user behavior data, rather than ad-hoc heuristics. This exploratory project investigates machine learning approaches for constructing evaluation metrics for Web search and information retrieval that consider along important directions other than relevance such as diversity, balance and coverage. The approach is based on fundamentally extending the popular evaluation metric Discounted Cumulated Gains (DCG). Research focuses on developing optimization methods for learning DCG that can incorporate the degree of difference in pair-wise comparison of ranking lists. Machine learning methods that can learn DCG for the more realistic scenarios where the relevance grades are not readily available are explored, and nonlinear utility functions as evaluation metrics that can accurately capture the quality of search result sets in terms of relevance, diversity, coverage, balance and novelty are investigated.The project has a number of broad impacts. Research results are expected to provide foundations for further research in evaluation metrics. Active collaborations with industry leaders in Web search will enable the resulting methods to have real impacts on search engine as well as large IR system performance improvements. Improving the quality of search results will have significant impacts on satisfying people's information needs as well as their quality of life in general. The set of research topics lies at the interface between information retrieval and machine learning applications and it provides an ideal setting for training undergraduate and graduate students in the emerging interdisciplinary field of Web of science and engineering research. The project Web site (http://www.cc.gatech.edu/~zha/metrics.html) will be used for results dissemination.