Predictive Modeling of User Performance for Validated Effectiveness Measures of Search Engine Quality

用户表现的预测建模，用于验证搜索引擎质量的有效性指标

• 批准号: RGPIN-2014-03642
• 负责人: Smucker, Mark
• 金额: $ 2.33万
• 依托单位: University of Waterloo
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2016
• 资助国家: 加拿大
• 起止时间: 2016-01-01 至 2017-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=610563
• 关键词: Predictive; Modeling; User; Performance; Validated

Effectiveness measures play a critical role in the development of new search engine algorithms. The effectiveness measures in wide use today have little or no explicit model of user behavior, no model of the user interface, and no estimate of the time it takes a user to find relevant information --- all user actions are falsely assumed to take the same amount of time and all users are falsely assumed to behave the same. As a result, these measures only produce scores that are correlated with user performance. In a recent breakthrough, we have produced an effectiveness measure that explicitly models the user, the user interface, and is calibrated with the time user actions take. This new effectiveness measure, time-biased gain, directly produces validated estimates of user performance and can even estimate the distribution of performance by modeling the different abilities and probabilistic actions of users. My research program studies user behavior with search engines and then tries to model this behavior for use in new effectiveness measures, as well as, to help produce new ways of interacting with the user to improve people's ability to satisfy their information needs. The next 5 years of my research program will focus on the further development of effectiveness measures with a long term objective of making the evaluation of search engine quality synonymous with the estimation of user performance. Previous to the results of my research program, user performance was primarily estimated through expensive and rarely conducted laboratory user studies with human participants; the developers of new retrieval algorithms were required to optimize their methods against the proxy values produced by existing measures. Specific objectives of my research program include i) accurately modeling multiple-query sessions, ii) creating better models of the time spent searching, and iii) making time-biased gain and other validated effectiveness measures a complete solution for researchers and engineers. The impact of this research will be widely felt and far reaching. Search engines have become an integral part of Canada's knowledge-based economy. We use search engines to help us in nearly endless ways. Engineers can find answers to their detailed questions in minutes rather than days, doctors as well as patients use search engines to research medical treatments for life-threatening illnesses, and lawyers depend on search engines to find past cases with legal e-discovery becoming an industry in itself. Better effectiveness measures, where better means more accurate predictions of user performance, will allow researchers and engineers to speed the rate at which they improve retrieval algorithms. In building better effectiveness measures, we will produce fundamental knowledge about human information processing as well the decision making processes of humans when faced with uncertainty, and this knowledge should have application in the fields of psychology, management sciences, and of course, the improvement of other knowledge-based software applications. The proposed research program will provide ample opportunities for students to gain experience in high value areas such as modeling and prediction of human behavior as well as large scale data processing.

有效性度量在开发新的搜索引擎算法中起着至关重要的作用。目前广泛使用的有效性度量很少或根本没有明确的用户行为模型，没有用户界面模型，也没有对用户查找相关信息所需时间的估计——所有用户操作都被错误地假设为花费相同的时间，所有用户的行为都被错误地假设为相同。因此，这些措施只产生与用户表现相关的分数。在最近的一项突破中，我们已经产生了一种有效的度量，可以明确地对用户、用户界面进行建模，并根据用户操作所花费的时间进行校准。这种新的有效性度量，即时间偏差增益，直接产生对用户性能的有效估计，甚至可以通过对用户的不同能力和概率行为进行建模来估计性能的分布。我的研究项目是研究使用搜索引擎的用户行为，然后尝试将这种行为建模，用于新的有效性度量，以及帮助产生与用户交互的新方法，以提高人们满足信息需求的能力。我未来5年的研究计划将集中于进一步发展有效性措施，其长期目标是使搜索引擎质量的评估与用户性能的估计同义。在我的研究项目取得成果之前，用户性能主要是通过昂贵且很少进行的实验室用户研究来评估的；新检索算法的开发人员需要根据现有度量产生的代理值来优化他们的方法。我的研究计划的具体目标包括：1)准确地建模多个查询会话；2)创建更好的搜索时间模型；3)为研究人员和工程师提供时间偏差增益和其他经过验证的有效性度量的完整解决方案。这项研究的影响将是广泛而深远的。搜索引擎已经成为加拿大知识型经济不可或缺的一部分。我们使用搜索引擎以几乎无穷无尽的方式帮助我们。工程师可以在几分钟而不是几天内找到详细问题的答案，医生和病人使用搜索引擎研究危及生命的疾病的医学治疗方法，律师依靠搜索引擎查找过去的案例，法律电子发现本身已经成为一个行业。更好的有效性度量——更好意味着对用户表现的更准确预测——将允许研究人员和工程师加快他们改进检索算法的速度。在建立更好的有效性措施时，我们将产生关于人类信息处理以及人类面对不确定性时的决策过程的基本知识，这些知识应该在心理学，管理科学领域中得到应用，当然，还有其他基于知识的软件应用程序的改进。拟议的研究计划将为学生提供充足的机会获得高价值领域的经验，如人类行为的建模和预测以及大规模数据处理。