Predictive Modeling of User Performance for Validated Effectiveness Measures of Search Engine Quality

有效性度量在新搜索引擎算法的开发中起着至关重要的作用。如今广泛使用的有效性度量很少或根本没有明确的用户行为模型，没有用户界面模型，也没有估计用户找到相关信息所需的时间-错误地假设所有用户操作花费的时间相同，并且错误地假设所有用户的行为都相同。因此，这些衡量标准只产生与用户表现相关的分数。在最近的一项突破中，我们提出了一种有效性度量，它明确地对用户、用户界面进行建模，并根据用户操作所花费的时间进行校准。这种新的有效性度量，时间偏置增益，直接产生用户性能的有效估计，甚至可以通过对用户的不同能力和概率行为建模来估计性能的分布。我的研究项目通过搜索引擎研究用户行为，然后尝试对这种行为进行建模，以用于新的有效性衡量标准，并帮助产生与用户互动的新方式，以提高人们满足信息需求的能力。在接下来的5年里，我的研究计划将专注于进一步开发有效性衡量标准，长期目标是使搜索引擎质量评估与用户表现评估成为同义词。在我的研究项目结果出来之前，用户表现主要是通过昂贵的、很少与人类参与者进行的实验室用户研究来估计的；新检索算法的开发者被要求根据现有测量产生的代理值来优化他们的方法。我的研究计划的具体目标包括：i)准确地模拟多个查询会话；ii)创建更好的搜索时间模型；iii)为研究人员和工程师提供一个完整的解决方案：基于时间的增益和其他经过验证的有效性衡量标准。这项研究的影响将是广泛和深远的。搜索引擎已经成为加拿大知识经济不可或缺的一部分。我们使用搜索引擎以几乎无穷无尽的方式帮助我们。工程师可以在几分钟内找到他们详细问题的答案，而不是几天，医生和患者使用搜索引擎来研究危及生命的疾病的医疗治疗，律师依靠搜索引擎来查找过去的案件，法律电子证据开示本身就成为了一个行业。更好的有效性衡量标准意味着对用户表现的更准确预测，这将使研究人员和工程师加快他们改进检索算法的速度。在建立更好的有效性措施时，我们将产生关于人类信息处理以及人类在面临不确定性时的决策过程的基本知识，这些知识应该应用于心理学、管理科学，当然还有其他基于知识的软件应用程序的改进。拟议的研究计划将为学生提供充足的机会，在高价值领域获得经验，如人类行为的建模和预测以及大规模数据处理。

Effectiveness measures play a critical role in the development of new search engine algorithms. The effectiveness measures in wide use today have little or no explicit model of user behavior, no model of the user interface, and no estimate of the time it takes a user to find relevant information --- all user actions are falsely assumed to take the same amount of time and all users are falsely assumed to behave the same. As a result, these measures only produce scores that are correlated with user performance. In a recent breakthrough, we have produced an effectiveness measure that explicitly models the user, the user interface, and is calibrated with the time user actions take. This new effectiveness measure, time-biased gain, directly produces validated estimates of user performance and can even estimate the distribution of performance by modeling the different abilities and probabilistic actions of users. My research program studies user behavior with search engines and then tries to model this behavior for use in new effectiveness measures, as well as, to help produce new ways of interacting with the user to improve people's ability to satisfy their information needs. The next 5 years of my research program will focus on the further development of effectiveness measures with a long term objective of making the evaluation of search engine quality synonymous with the estimation of user performance. Previous to the results of my research program, user performance was primarily estimated through expensive and rarely conducted laboratory user studies with human participants; the developers of new retrieval algorithms were required to optimize their methods against the proxy values produced by existing measures. Specific objectives of my research program include i) accurately modeling multiple-query sessions, ii) creating better models of the time spent searching, and iii) making time-biased gain and other validated effectiveness measures a complete solution for researchers and engineers. The impact of this research will be widely felt and far reaching. Search engines have become an integral part of Canada's knowledge-based economy. We use search engines to help us in nearly endless ways. Engineers can find answers to their detailed questions in minutes rather than days, doctors as well as patients use search engines to research medical treatments for life-threatening illnesses, and lawyers depend on search engines to find past cases with legal e-discovery becoming an industry in itself. Better effectiveness measures, where better means more accurate predictions of user performance, will allow researchers and engineers to speed the rate at which they improve retrieval algorithms. In building better effectiveness measures, we will produce fundamental knowledge about human information processing as well the decision making processes of humans when faced with uncertainty, and this knowledge should have application in the fields of psychology, management sciences, and of course, the improvement of other knowledge-based software applications. The proposed research program will provide ample opportunities for students to gain experience in high value areas such as modeling and prediction of human behavior as well as large scale data processing.