Questionnaire mode effects in interactive information retrieval experiments
Questionnaire mode effects in interactive information retrieval experiments
复制标题
DOI：
10.1016/j.ipm.2007.02.007
复制
发表时间：
2008-01-01
影响因子：
8.6
通讯作者：
Landau, Brian
中科院分区：
文献类型：
--
作者：
Kelly, Diane;Harper, David J.;Landau, Brian
在交互式信息检索实验中，问卷调查是从被试中收集数据的重要手段。调查方法学、民意测验和心理学的研究表明，受试者在回答问卷时会表现出一些反应偏差和行为。此外，人机交互的研究表明，受试者在完成可用性问卷时倾向于夸大他们对系统的评级。在这项研究中，我们调查问卷模式和受试者的可用性问卷调查的封闭式和开放式的问题，在互动IR实验管理的反应之间的关系。采用纸笔、电子和访谈三种调查方式，对51名使用两种信息检索系统的被试进行了问卷调查。结果表明，访谈模式下被试对系统的定量评价显著低于电子模式。关于开放式问题，在采访模式中的主题使用了显着更多的话比主题的笔和纸或电子模式来传达他们的反应，并传达了显着更高数量的响应单元，即使在整个条件下的独特的响应单元的总数大致相同。最后，结果表明，在笔和纸的模式是最有效的沟通，他们的开放性问题的答案。这些结果表明，研究人员应该使用访谈模式，以引起封闭的问题，从主题的反应，无论是笔和纸或电子模式，以引起开放性问题的反应。(c)2007年由Elsevier Ltd.出版
The questionnaire is an important technique for gathering data from subjects during interactive information retrieval (IR) experiments. Research in survey methodology, public opinion polling and psychology has demonstrated a number of response biases and behaviors that subjects exhibit when responding to questionnaires. Furthermore, research in human-computer interaction has demonstrated that subjects tend to inflate their ratings of systems when completing usability questionnaires. In this study we investigate the relationship between questionnaire mode and subjects' responses to a usability questionnaire comprised of closed and open questions administered during an interactive IR experiment. Three questionnaire modes (pen-and-paper, electronic and interview) were explored with 51 subjects who used one of two information retrieval systems. Results showed that subjects' quantitative evaluations of systems were significantly lower in the interview mode than in the electronic mode. With respect to open questions, subjects in the interview mode used significantly more words than subjects in the pen-and-paper or electronic modes to communicate their responses, and communicated a significantly higher number of response units, even though the total number of unique response units was roughly the same across condition. Finally, results showed that subjects in the pen-and-paper mode were the most efficient in communicating their responses to open questions. These results suggest that researchers should use the interview mode to elicit responses to closed questions from subjects and either pen-and-paper or electronic modes to elicit responses to open questions. (c) 2007 Published by Elsevier Ltd.