CAREER: Measuring Search Engines' Ability to Help Users Complete Tasks

这个项目的目的是提高搜索系统帮助用户完成任务的能力。任何搜索引擎的有用性最终取决于它在帮助用户方面有多好。系统及其使用的任务可能非常复杂；系统实现或任务执行中的微小变化可能会对系统的有用性产生重大影响，特别是在大量人员使用的较长寿命内。理解效用的传统方法涉及使用测试集合，测试集合包括要搜索的文档的集合、不变的信息需求以及对文档与需求的相关性的人工判断；这些组件被放入一个简单的批处理过程中，该过程测量搜索效率并测试简单的统计假设。虽然这种方法很有用，但它通常无法捕获用户和任务中存在的可变性：不同的用户经常以非常不同的方式与同一系统交互，这意味着对一个用户或一个任务有用的系统可能对另一个用户或任务没有用。因此，这个项目的重点是开发新的方法来理解、评估和改进考虑可变性的信息检索(IR)系统的有用性。本项目中研究的方法旨在模拟用户与系统的交互以完成任务，包括用户如何确定上下文中的相关性、他们如何随着时间的推移修改他们与系统的交互，以及不同用户的不同方法如何影响整个系统的有用性。该项目将为基于批次系统的信息检索评估产生新型的测试集合、评估措施和统计方法，供学术界和工业界的研究人员和从业人员使用。这项工作将演示如何利用这两者来提高对大多数用户的系统效用，并提出关于信息检索系统开发中的因果关系的更深层次的假设，从而导致信息检索技术在所有领域的改进。研究将与学生以及研究人员和实践者的教育活动相结合，以学习先进的实验设计和分析。教育工作将包括IR和计算机科学中经验方法的教程和教学课程，更广泛的科学界使用的方法，以及新开发的方法与这些方法的关系。本项目产生的结果可在项目网站(http://ir.cis.udel.edu/IIS-1350799).)上找到

The purpose of this project is to improve search systems' ability to help users complete tasks. The usefulness of any search engine ultimately depends on how good it is at aiding its users. The systems and the tasks they are used for can be very complicated; small changes in a system's implementation or a task's execution can have major effects on the usefulness of the system, especially over a long lifespan of use by a large base of people. The traditional approach to understanding utility involves the use of test collections, which consist of a collection of documents to be searched, unchanging information needs, and human judgments of the relevance of documents to needs; these components are put into a simple batch process that measures search effectiveness and tests simple statistical hypotheses. While this approach is useful, it often fails to capture variability present in users and tasks: different users often interact with the same system in very different ways, meaning a system that is useful for one user or one task may not be useful for another user or task. Therefore, this project focuses on developing new methods for understanding, estimating, and improving the usefulness of information retrieval (IR) systems that take variability into consideration. The methods investigated in this project are designed to model user interactions with a system to complete a task, including how users determine relevance in context, how they modify their interaction with a system over time, and how different approaches by different users affect the overall system usefulness. The project will produce new types of test collections, evaluation measures, and statistical methods for batch-style systems-based information retrieval evaluation for use by researchers and practitioners in academia and industry. The work will demonstrate how to use these both to improve system utility to a population of users as well as to pose deeper hypotheses about causality in IR system development, thus leading to improvements in IR technology in all domains. Research will be integrated with educational activities for students as well as researchers and practitioners to learn advanced experimental design and analysis. Educational efforts will include tutorials and teaching courses on empirical methods in IR and computer science, methods in use in the wider scientific community, and how the newly developed methods relate to those. Results produced from this project can be found on the project web site (http://ir.cis.udel.edu/IIS-1350799).