User behavior models for information access evaluation

点击翻译按钮获取中文摘要

To improve search engines, and other information access systems, their performance must be measured in human terms. A key measure is the time taken to find required information and to complete the task at hand. Whether a user is searching for a single fact, or is researching a complex problem, the user wants to satisfy their information as fast as possible. Both researchers and engineers need a means to quickly and affordably estimate the rate at which users can satisfy their information needs with a given search engine. To produce accurate estimates of user performance, effectiveness measures must be designed that can adjust to different user tasks and computing devices. For example, users searching for a simple fact are expected to behave differently than users researching a complex problem. Likewise, the device used to access information can have a significant impact on how a user behaves. For mobile search, device constraints such as a small screen and a virtual keyboard need to be taken into consideration. Traditional effectiveness measures are ill-suited to estimate user effectiveness across different user tasks and access devices given their lack of user behavior models, which prevents adapting them to the search context. This work proposes to further develop the time-biased gain effectiveness measure, which can be adapted to the search context.The research undertaken by this project builds on existing work by Professors Smucker and Clarke. This existing work proposes and develops time-biased gain (TBG) effectiveness measures for evaluating search engine performance. Unlike traditional measures of search engine performance, TBG makes time a key part of the measure. Computations of TBG are calibrated to times and decision processes extracted from usage logs and user studies. As changes are made to a search engine's ranking algorithms, interface, and other components, TBG may be used to predict the impact of these changes on user performance. In order to advance this work to more complex scenarios and domains, the research will incorporate insights uncovered by Professor Cormack into relevance prediction. NSERC funding under this project serves to extend an existing Google Focus Award on context-aware mobile social networking. While developing technologies under that proposal, a need for improved evaluation methodologies became apparent. In addition to supporting the needs of this Google Focus Award, this project advances our general understanding of information access evaluation.