User behavior models for information access evaluation

为了改进搜索引擎和其他信息获取系统，它们的表现必须用人的标准来衡量。一个关键的衡量标准是查找所需信息和完成手头任务所需的时间。无论用户是在搜索单个事实，还是在研究复杂的问题，用户都希望尽可能快地满足他们的信息。研究人员和工程师都需要一种方法来快速且负担得起地估计用户使用给定搜索引擎满足其信息需求的速度。为了准确估计用户性能，必须设计能够适应不同用户任务和计算设备的有效性衡量标准。例如，搜索简单事实的用户应该与研究复杂问题的用户表现不同。同样，用于访问信息的设备可能会对用户的行为产生重大影响。对于移动搜索，需要考虑小屏幕和虚拟键盘等设备限制。由于缺乏用户行为模型，传统的效率指标不适合评估不同用户任务和访问设备的用户效率，这阻碍了它们适应搜索环境。这项工作建议进一步发展时间偏置的增益有效性度量，该度量可以适应搜索环境。 该项目开展的研究建立在斯莫克教授和克拉克教授现有工作的基础上。这项已有的工作提出并开发了用于评估搜索引擎性能的时间偏置增益(TBG)有效性度量。与传统的搜索引擎性能衡量标准不同，TBG将时间作为衡量标准的关键部分。根据从使用日志和用户研究中提取的时间和决策过程来校准TBG的计算。当搜索引擎的排名算法、界面和其他组件发生变化时，可以使用TBG来预测这些变化对用户性能的影响。为了将这项工作推进到更复杂的场景和领域，这项研究将把科马克教授发现的见解纳入相关性预测。 NSERC在该项目下的资金用于扩展现有的Google Focus奖，该奖项旨在支持情景感知的移动社交网络。在根据该提案开发技术的同时，显然需要改进评价方法。除了支持本次Google Focus奖的需求外，该项目还促进了我们对信息获取评估的总体理解。

To improve search engines, and other information access systems, their performance must be measured in human terms. A key measure is the time taken to find required information and to complete the task at hand. Whether a user is searching for a single fact, or is researching a complex problem, the user wants to satisfy their information as fast as possible. Both researchers and engineers need a means to quickly and affordably estimate the rate at which users can satisfy their information needs with a given search engine. To produce accurate estimates of user performance, effectiveness measures must be designed that can adjust to different user tasks and computing devices. For example, users searching for a simple fact are expected to behave differently than users researching a complex problem. Likewise, the device used to access information can have a significant impact on how a user behaves. For mobile search, device constraints such as a small screen and a virtual keyboard need to be taken into consideration. Traditional effectiveness measures are ill-suited to estimate user effectiveness across different user tasks and access devices given their lack of user behavior models, which prevents adapting them to the search context. This work proposes to further develop the time-biased gain effectiveness measure, which can be adapted to the search context. The research undertaken by this project builds on existing work by Professors Smucker and Clarke. This existing work proposes and develops time-biased gain (TBG) effectiveness measures for evaluating search engine performance. Unlike traditional measures of search engine performance, TBG makes time a key part of the measure. Computations of TBG are calibrated to times and decision processes extracted from usage logs and user studies. As changes are made to a search engine's ranking algorithms, interface, and other components, TBG may be used to predict the impact of these changes on user performance. In order to advance this work to more complex scenarios and domains, the research will incorporate insights uncovered by Professor Cormack into relevance prediction. NSERC funding under this project serves to extend an existing Google Focus Award on context-aware mobile social networking. While developing technologies under that proposal, a need for improved evaluation methodologies became apparent. In addition to supporting the needs of this Google Focus Award, this project advances our general understanding of information access evaluation.