Relevance Assessment Tool (RAT): Development and enhancement of a software for conducting studies based on results from search systems

相关性评估工具 (RAT)：开发和增强用于根据搜索系统结果进行研究的软件

• 批准号: 460676551
• 负责人: Professor Dr. Dirk Lewandowski
• 金额: --
• 依托单位: Department Information
• 依托单位国家: 德国
• 项目类别: Research data and software (Scientific Library Services and Information Systems)
• 资助国家: 德国
• 项目状态: 未结题
• 来源: https://gepris.dfg.de/gepris/projekt/460676551?language=en
• 关键词: Relevance; Assessment; Tool; RAT; Development

The project's overall goal is to continue to develop and enhance the Relevance Assessment Tool (RAT) so that it will become a stable, flexible and sustainable tool for conducting studies that deal with the collection and analysis of data from search engines and other information retrieval systems.The Relevance Assessment Tool (RAT) is a software toolkit that allows researchers to conduct large-scale studies based on results from (commercial) search engines and other information retrieval systems. It consists of modules for (1) designing studies, (2) collecting data from search systems, (3) collecting judgments on the results, (4) downloading/analysing the results. A starting point to developing RAT was the fact that retrieval effectiveness studies usually require much manual work in designing the test, collecting search results, finding jurors and collecting their assessments, and in analysing the test results, as well. RAT addresses all these issues and aims to offer help in making retrieval studies more efficient and effective.The design of the RAT prototype has been guided by the requirement of researchers to query external information retrieval systems in general, and search engines in particular. This need derives from researchers' interest in the quality of their search results, the composition of their results (lists or pages), and potential biases in the results, to name but a few. This is a unique approach that separates RAT from software developed to aid retrieval evaluation where researchers have complete control over the systems evaluated.RAT allows studies to be conducted under the two major evaluation paradigms, namely system-centred and user-centred evaluations. RAT will also be useful to any other researchers aiming to use results from search engines for further analysis. While the basis of RAT lies in information retrieval evaluation, its use goes well beyond this research area.Already in the prototype phase, we found that there is a significant need for a software tool like RAT in the search engine and information retrieval communities, respectively. Some projects have already demonstrated this.In the dissemination of the software, we will first focus on the search engine research/information retrieval communities, and then roll out the software further. To address these communities, we will (1) have the software development accompanied by some pilot test users; (2) promote the software through conferences, an advisory board, and publications. To target the intended researchers, we aim to make the software well usable also to researchers who are not in the field of information science or computer science. All software developed will be made available open source.We will ensure the sustainability of the project results through measures in three areas: (1) Establishing and distributing the software, (2) establishing and maintaining a user and developer community, (3) publishing the software open source.

该项目的总体目标是继续开发和加强相关性评估工具，相关性评估工具（Relevance Assessment Tool，RAT）是一个软件工具包，允许研究人员根据搜索引擎和其他信息检索系统的结果进行大规模研究。（商业）搜索引擎和其他信息检索系统。它由以下模块组成：（1）设计研究，（2）从搜索系统收集数据，（3）收集对结果的判断，（4）下载/分析结果。开发检索系统的一个出发点是，检索有效性研究通常需要在设计测试、收集搜索结果、寻找陪审员和收集他们的评估以及分析测试结果方面进行大量的手工工作。RAT解决了所有这些问题，并旨在提供帮助，使检索研究更有效和有效的RAT原型的设计已被研究人员查询外部信息检索系统的一般要求，特别是搜索引擎的指导。这种需求源于研究人员对搜索结果的质量、结果的组成（列表或页面）以及结果中的潜在偏见的兴趣，仅举几例。这是一种独特的方法，将RAT与为辅助检索评价而开发的软件分开，研究人员可以完全控制所评价的系统，RAT允许在两种主要评价范式下进行研究，即以系统为中心的评价和以用户为中心的评价。RAT也将是有用的任何其他研究人员旨在使用搜索引擎的结果进行进一步分析。虽然RAT的基础在于信息检索评估，它的使用远远超出了这一研究领域。已经在原型阶段，我们发现，有一个显着的需要，如RAT在搜索引擎和信息检索社区，分别软件工具。在推广软件时，我们会先集中在搜索引擎研究/资料检索界，然后再进一步推广软件。为了解决这些社区问题，我们将（1）在软件开发的同时配备一些试点测试用户;（2）通过会议、咨询委员会和出版物推广软件。为了针对预期的研究人员，我们的目标是使软件也很好地适用于信息科学或计算机科学领域以外的研究人员。所有开发的软件都将开放源码。我们将通过三个方面的措施确保项目成果的可持续性：（1）建立和分发软件，（2）建立和维护用户和开发者社区，（3）发布开源软件。