III: Small: Collection Construction Methodologies for Learning-to-Rank

现代搜索引擎，尤其是为万维网设计的搜索引擎，通常分析并联合收割机组合从所提交的查询和底层文档（例如，网页），以便评估文档与给定查询的相对相关性，从而对底层集合进行排名。这个问题的庞大规模已经导致了学习排名算法的发展，该算法可以自动构建这样的排名函数：给定（特征向量，相关性）对的训练集，机器学习过程学习如何以这样的方式联合收割机组合查询和文档特征，以便有效地评估任何文档与任何查询的相关性，从而响应于用户输入对集合进行排名。许多思想和研究已经放在特征提取和复杂的学习排名算法的发展。然而，相对较少的研究已经进行了学习排名数据集的文件和查询的选择，也没有对这些选择的学习排名算法的能力的影响，有效地和efficient.The建议的工作调查的查询，文件和功能选择的学习排名算法的能力，有效地和有效地学习排名功能的影响。 在文档选择的初步结果中，一项试点研究已经确定，大小仅为通常使用的训练集的2%至5%的训练集对于学习排名目的同样有效。因此，人们可以在更小的数据集上更有效地训练（尽管实际上是等效的），或者以相同的成本，人们可以在更大的和更有代表性的数据集上训练。 除了正式描述这种现象用于文档选择之外，拟议的工作还研究了这种现象用于查询和特征选择，最终目标是（1）了解文档、查询的影响，和特征选择的学习排名算法和（2）开发收集建设的方法，是高效和有效的学习，除了描述和开发馆藏建设方法外，项目计划还包括开发和发布新的、高效的、有效的学习排名数据集，供学术界和工业界使用。 在促进这一努力，项目组与国家标准与技术研究所（NIST）和微软研究院，两个主要的组织，开发和发布信息检索数据集的密切联系。 作为该项目一部分开发的所有研究成果和数据集将在项目网站上公布（http：//www.ccs.neu.edu/home/jaa/IIS-1017903/）。该项目为学生提供教育和培训经验。

Modern search engines, especially those designed for the World Wide Web, commonly analyze and combine hundreds of features extracted from the submitted query and underlying documents (e.g., web pages) in order to assess the relative relevance of a document to a given query and thus rank the underlying collection. The sheer size of this problem has led to the development of learning-to-rank algorithms that can automate the construction of such ranking functions: Given a training set of (feature vector, relevance) pairs, a machine learning procedure learns how to combine the query and document features in such a way so as to effectively assess the relevance of any document to any query and thus rank a collection in response to a user input. Much thought and research has been placed on feature extraction and the development of sophisticated learning-to-rank algorithms. However, relatively little research has been conducted on the choice of documents and queries for learning-to-rank data sets nor on the effect of these choices on the ability of a learning-to-rank algorithm to "learn", effectively and efficiently.The proposed work investigates the effect of query, document, and feature selection on the ability of learning-to-rank algorithms to efficiently and effectively learn ranking functions. In preliminary results on document selection, a pilot study has already determined that training sets whose sizes are as small as 2 to 5% of those typically used are just as effective for learning-to-rank purposes. Thus, one can train more efficiently over a much smaller (though effectively equivalent) data set, or, at an equal cost, one can train over a far "larger" and more representative data set. In addition to formally characterizing this phenomenon for document selection, the proposed work investigate this phenomenon for query and feature selection as well, with the end goals of (1) understanding the effect of document, query, and feature selection on learning-to-rank algorithms and (2) developing collection construction methodologies that are efficient and effective for learning-to-rank purposes.In addition to characterizing and developing collection construction methodologies, the project plan includes development and release of new, efficient, and effective learning-to-rank data sets for use by academia and industry. In fostering this effort, the project team has close ties with the National Institute of Standards and Technology (NIST) and Microsoft Research, two of the premier organizations that develop and release Information Retrieval data sets. All research results and data sets developed as part of this project will be made available at the project website (http://www.ccs.neu.edu/home/jaa/IIS-1017903/). The project provides an educational and training experience for students.