Relevance Assessment Tool (RAT): Development and enhancement of a software for conducting studies based on results from search systems

该项目的总体目标是继续开发和加强相关性评估工具(RAT)，使其成为一个稳定、灵活和可持续的工具，用于进行研究，收集和分析来自搜索引擎和其他信息检索系统的数据。相关性评估工具(RAT)是一个软件工具包，允许研究人员根据(商业)搜索引擎和其他信息检索系统的结果进行大规模研究。它包括(1)设计研究、(2)从搜索系统收集数据、(3)收集对结果的判断、(4)下载/分析结果的模块。开发RAT的一个起点是，检索效果研究通常需要在设计测试、收集搜索结果、寻找陪审员和收集他们的评估以及分析测试结果方面进行大量人工工作。RAT解决了所有这些问题，旨在帮助提高检索研究的效率和效果。RAT原型的设计是根据研究人员对外部信息检索系统，特别是搜索引擎的查询需求而设计的。这一需求源于研究人员对其搜索结果的质量、结果(列表或页面)的组成以及结果中的潜在偏见的兴趣，仅举几个例子。这是一种独特的方法，将RAT与为辅助检索评估而开发的软件分开，在该软件中，研究人员可以完全控制被评估的系统。RAT允许在两种主要评估范式下进行研究，即以系统为中心的评估和以用户为中心的评估。RAT也将对任何其他希望利用搜索引擎的结果进行进一步分析的研究人员有用。虽然RAT的基础在于信息检索评价，但它的应用远远超出了这一研究领域。我们已经在原型阶段发现，在搜索引擎和信息检索领域，分别对RAT这样的软件工具有很大的需求。一些项目已经证明了这一点。在软件的分发方面，我们将首先专注于搜索引擎研究/信息检索社区，然后再进一步推出该软件。为了解决这些社区的问题，我们将(1)在一些试点测试用户的陪同下进行软件开发；(2)通过会议、顾问委员会和出版物来推广软件。为了针对预期的研究人员，我们的目标是使该软件对非信息科学或计算机科学领域的研究人员也能很好地使用。我们将通过三个方面的措施确保项目成果的可持续性：(1)建立和分发软件，(2)建立和维护用户和开发人员社区，(3)发布软件开源。

The project's overall goal is to continue to develop and enhance the Relevance Assessment Tool (RAT) so that it will become a stable, flexible and sustainable tool for conducting studies that deal with the collection and analysis of data from search engines and other information retrieval systems.The Relevance Assessment Tool (RAT) is a software toolkit that allows researchers to conduct large-scale studies based on results from (commercial) search engines and other information retrieval systems. It consists of modules for (1) designing studies, (2) collecting data from search systems, (3) collecting judgments on the results, (4) downloading/analysing the results. A starting point to developing RAT was the fact that retrieval effectiveness studies usually require much manual work in designing the test, collecting search results, finding jurors and collecting their assessments, and in analysing the test results, as well. RAT addresses all these issues and aims to offer help in making retrieval studies more efficient and effective.The design of the RAT prototype has been guided by the requirement of researchers to query external information retrieval systems in general, and search engines in particular. This need derives from researchers' interest in the quality of their search results, the composition of their results (lists or pages), and potential biases in the results, to name but a few. This is a unique approach that separates RAT from software developed to aid retrieval evaluation where researchers have complete control over the systems evaluated.RAT allows studies to be conducted under the two major evaluation paradigms, namely system-centred and user-centred evaluations. RAT will also be useful to any other researchers aiming to use results from search engines for further analysis. While the basis of RAT lies in information retrieval evaluation, its use goes well beyond this research area.Already in the prototype phase, we found that there is a significant need for a software tool like RAT in the search engine and information retrieval communities, respectively. Some projects have already demonstrated this.In the dissemination of the software, we will first focus on the search engine research/information retrieval communities, and then roll out the software further. To address these communities, we will (1) have the software development accompanied by some pilot test users; (2) promote the software through conferences, an advisory board, and publications. To target the intended researchers, we aim to make the software well usable also to researchers who are not in the field of information science or computer science. All software developed will be made available open source.We will ensure the sustainability of the project results through measures in three areas: (1) Establishing and distributing the software, (2) establishing and maintaining a user and developer community, (3) publishing the software open source.