User behavior models for information access evaluation

为了改进搜索引擎和其他信息访问系统，它们的性能必须以人为本来衡量。一个关键的度量是查找所需信息和完成手头任务所花费的时间。无论用户是在搜索单个事实，还是在研究一个复杂的问题，用户都希望尽可能快地满足他们的信息。研究人员和工程师都需要一种方法来快速、经济地估计用户在给定搜索引擎上满足信息需求的速度。为了准确估计用户性能，必须设计能够适应不同用户任务和计算设备的有效性度量。例如，搜索简单事实的用户的行为与搜索复杂问题的用户的行为不同。同样，用于访问信息的设备也会对用户的行为产生重大影响。对于移动搜索，需要考虑小屏幕和虚拟键盘等设备限制。由于缺乏用户行为模型，传统的有效性度量不适合评估跨不同用户任务和访问设备的用户有效性，这使得它们无法适应搜索上下文。这项工作提出了进一步发展时偏增益有效性度量，它可以适应搜索上下文。

To improve search engines, and other information access systems, their performance must be measured in human terms. A key measure is the time taken to find required information and to complete the task at hand. Whether a user is searching for a single fact, or is researching a complex problem, the user wants to satisfy their information as fast as possible. Both researchers and engineers need a means to quickly and affordably estimate the rate at which users can satisfy their information needs with a given search engine. To produce accurate estimates of user performance, effectiveness measures must be designed that can adjust to different user tasks and computing devices. For example, users searching for a simple fact are expected to behave differently than users researching a complex problem. Likewise, the device used to access information can have a significant impact on how a user behaves. For mobile search, device constraints such as a small screen and a virtual keyboard need to be taken into consideration. Traditional effectiveness measures are ill-suited to estimate user effectiveness across different user tasks and access devices given their lack of user behavior models, which prevents adapting them to the search context. This work proposes to further develop the time-biased gain effectiveness measure, which can be adapted to the search context. The research undertaken by this project builds on existing work by Professors Smucker and Clarke. This existing work proposes and develops time-biased gain (TBG) effectiveness measures for evaluating search engine performance. Unlike traditional measures of search engine performance, TBG makes time a key part of the measure. Computations of TBG are calibrated to times and decision processes extracted from usage logs and user studies. As changes are made to a search engine's ranking algorithms, interface, and other components, TBG may be used to predict the impact of these changes on user performance. In order to advance this work to more complex scenarios and domains, the research will incorporate insights uncovered by Professor Cormack into relevance prediction. NSERC funding under this project serves to extend an existing Google Focus Award on context-aware mobile social networking. While developing technologies under that proposal, a need for improved evaluation methodologies became apparent. In addition to supporting the needs of this Google Focus Award, this project advances our general understanding of information access evaluation.