III: Small: Collection Construction Methodologies for Learning-to-Rank

现代搜索引擎，尤其是那些为万维网设计的搜索引擎，通常分析和组合从提交的查询和底层文档(例如，网页)中提取的数百个特征，以便评估文档与给定查询的相对相关性，从而对底层集合进行排名。这个问题的巨大规模已经导致了能够自动构建这样的排序函数的学习排序算法的发展：给定(特征向量、相关性)对的训练集，机器学习过程学习如何以这样的方式组合查询和文档特征，以便有效地评估任何文档与任何查询的相关性，从而响应于用户输入来对集合进行排序。人们在特征提取和复杂的学习排序算法的开发上投入了大量的思考和研究。然而，对于学习排序数据集的文档和查询的选择以及这些选择对学习排序算法的能力的影响的研究相对较少，本文研究了查询、文档和特征选择对学习排序算法高效学习排序函数能力的影响。在关于文件选择的初步结果中，一项试点研究已经确定，其大小仅为通常使用的训练集的2%至5%的训练集对于学习到排名的目的同样有效。因此，一个人可以在一个小得多的(尽管实际上是等价的)数据集上更有效地训练，或者，在相同的成本下，一个人可以在一个“更大”和更具代表性的数据集上进行训练。除了形式化地描述这种现象用于文档选择之外，拟议的工作也研究这种现象用于查询和特征选择，最终目标是(1)了解文档、查询和特征选择对学习排序算法的影响，以及(2)开发用于学习排序目的的高效和有效的集合构建方法。除了表征和开发集合构建方法之外，该项目计划还包括开发和发布新的、高效和有效的学习排序数据集，供学术界和工业界使用。在推动这一努力的过程中，项目团队与国家标准与技术研究所(NIST)和微软研究院(Microsoft Research)保持着密切的联系，这两个组织是开发和发布信息检索数据集的主要组织。作为该项目的一部分开发的所有研究成果和数据集将在项目网站(http://www.ccs.neu.edu/home/jaa/IIS-1017903/).上提供该项目为学生提供了一种教育和培训体验。

Modern search engines, especially those designed for the World Wide Web, commonly analyze and combine hundreds of features extracted from the submitted query and underlying documents (e.g., web pages) in order to assess the relative relevance of a document to a given query and thus rank the underlying collection. The sheer size of this problem has led to the development of learning-to-rank algorithms that can automate the construction of such ranking functions: Given a training set of (feature vector, relevance) pairs, a machine learning procedure learns how to combine the query and document features in such a way so as to effectively assess the relevance of any document to any query and thus rank a collection in response to a user input. Much thought and research has been placed on feature extraction and the development of sophisticated learning-to-rank algorithms. However, relatively little research has been conducted on the choice of documents and queries for learning-to-rank data sets nor on the effect of these choices on the ability of a learning-to-rank algorithm to "learn", effectively and efficiently.The proposed work investigates the effect of query, document, and feature selection on the ability of learning-to-rank algorithms to efficiently and effectively learn ranking functions. In preliminary results on document selection, a pilot study has already determined that training sets whose sizes are as small as 2 to 5% of those typically used are just as effective for learning-to-rank purposes. Thus, one can train more efficiently over a much smaller (though effectively equivalent) data set, or, at an equal cost, one can train over a far "larger" and more representative data set. In addition to formally characterizing this phenomenon for document selection, the proposed work investigate this phenomenon for query and feature selection as well, with the end goals of (1) understanding the effect of document, query, and feature selection on learning-to-rank algorithms and (2) developing collection construction methodologies that are efficient and effective for learning-to-rank purposes.In addition to characterizing and developing collection construction methodologies, the project plan includes development and release of new, efficient, and effective learning-to-rank data sets for use by academia and industry. In fostering this effort, the project team has close ties with the National Institute of Standards and Technology (NIST) and Microsoft Research, two of the premier organizations that develop and release Information Retrieval data sets. All research results and data sets developed as part of this project will be made available at the project website (http://www.ccs.neu.edu/home/jaa/IIS-1017903/). The project provides an educational and training experience for students.