Reproducibility and simulation of interactive retrieval experiments (RESIRE)

点击翻译按钮获取中文摘要

Although reproducibility of experiments is a topic with increasing importance in information retrieval (IR), there has been hardly any work on reproducing interactive IR experiments. The main idea of our proposed project is to describe actual users by quantitative modeling of their behavior and their content-related decisions during retrieval sessions. These model parameters can then form the basis for comparing two interactive IR experiments with regard to reproducibility, and can also be used for simulating interactive IR, e. g. for evaluating new systems or system variants without (or before) carrying out actual user experiments. As part of the proposed project, we will define a new framework model for reproducibility of interactive IR experiments, develop new metrics for reproducibility and session quality, design new simulation methods integrating behavior and content-related decisions, develop an open source framework for reproduction and simulation of interactive IR, and perform extensive experiments using existing IR log data as well as observation data from new user experiments.