Predictive Modeling of User Performance for Validated Effectiveness Measures of Search Engine Quality

点击翻译按钮获取中文摘要

Effectiveness measures play a critical role in the development of new search engine algorithms. The effectiveness measures in wide use today have little or no explicit model of user behavior, no model of the user interface, and no estimate of the time it takes a user to find relevant information --- all user actions are falsely assumed to take the same amount of time and all users are falsely assumed to behave the same. As a result, these measures only produce scores that are correlated with user performance. In a recent breakthrough, we have produced an effectiveness measure that explicitly models the user, the user interface, and is calibrated with the time user actions take. This new effectiveness measure, time-biased gain, directly produces validated estimates of user performance and can even estimate the distribution of performance by modeling the different abilities and probabilistic actions of users. My research program studies user behavior with search engines and then tries to model this behavior for use in new effectiveness measures, as well as, to help produce new ways of interacting with the user to improve people's ability to satisfy their information needs. The next 5 years of my research program will focus on the further development of effectiveness measures with a long term objective of making the evaluation of search engine quality synonymous with the estimation of user performance. Previous to the results of my research program, user performance was primarily estimated through expensive and rarely conducted laboratory user studies with human participants; the developers of new retrieval algorithms were required to optimize their methods against the proxy values produced by existing measures. Specific objectives of my research program include i) accurately modeling multiple-query sessions, ii) creating better models of the time spent searching, and iii) making time-biased gain and other validated effectiveness measures a complete solution for researchers and engineers. The impact of this research will be widely felt and far reaching. Search engines have become an integral part of Canada's knowledge-based economy. We use search engines to help us in nearly endless ways. Engineers can find answers to their detailed questions in minutes rather than days, doctors as well as patients use search engines to research medical treatments for life-threatening illnesses, and lawyers depend on search engines to find past cases with legal e-discovery becoming an industry in itself. Better effectiveness measures, where better means more accurate predictions of user performance, will allow researchers and engineers to speed the rate at which they improve retrieval algorithms. In building better effectiveness measures, we will produce fundamental knowledge about human information processing as well the decision making processes of humans when faced with uncertainty, and this knowledge should have application in the fields of psychology, management sciences, and of course, the improvement of other knowledge-based software applications. The proposed research program will provide ample opportunities for students to gain experience in high value areas such as modeling and prediction of human behavior as well as large scale data processing.