Attention by design: Using attention checks to detect inattentive respondents and improve data quality
Attention by design: Using attention checks to detect inattentive respondents and improve data quality
复制标题
DOI：
10.1016/j.jom.2017.06.001
复制
发表时间：
2017-11-01
影响因子：
7.8
通讯作者：
Meloy, Margaret G.
中科院分区：
文献类型：
--
作者：
Abbey, James D.;Meloy, Margaret G.
本文研究了注意力检查和操纵验证，以检测在初级经验数据收集中注意力不集中的受访者。这些初级淡出注意力检查的范围从一个世纪前首次提出的简单的反向缩放等简单的方法，到更新、更复杂的方法，如通过在线数据捕获工具评估响应模式和定时响应。注意力检查验证的范围也从容易实现的机制，如通过定向查询进行自动检测，到研究人员对回答进行高度密集的调查。后者有可能导致无意中的研究人员偏见，因为研究人员的判断可能会影响对数据的解释。本研究的实证结果表明，结构和量表验证在匹配统计中显示出持续显著的改善--这一发现对于主要研究量表和结构的研究人员在他们的经验模型中非常有用。然而，根据分析中采用的基本实验模型，注意力检查通常不会显示出对实验操作的测试统计的重要性有一致的、系统的改善。后者的结果表明，就其本质而言，注意力检查可能会在失去样本受试者--权力降低和第二类错误增加--与仅利用机会的可能性之间触发一种内在的权衡--以前重要的结果实际上是第一类错误的结果。分析还表明，由于注意力检查而造成的流失率--在一些观察样本中高达70%--远远高于通常的假设。这样的失败率增加了一种幽灵，即没有验证注意力的研究可能会无意中增加他们的I型错误率。这份手稿提供了各种注意力检查的一般指导方针，讨论了方法的心理细微差别，并强调了激励一致性、金钱补偿和随后引发的受访者情绪之间的微妙平衡。(C)2017爱思唯尔B.V.保留所有权利。
This paper examines attention checks and manipulation validations to detect inattentive respondents in primary empirical data collection. These prima fade attention checks range from the simple such as reverse scaling first proposed a century ago to more recent and involved methods such as evaluating response patterns and timed responses via online data capture tools. The attention check validations also range from easily implemented mechanisms such as automatic detection through directed queries to highly intensive investigation of responses by the researcher. The latter has the potential to introduce inadvertent researcher bias as the researcher's judgment may impact the interpretation of the data. The empirical findings of the present work reveal that construct and scale validations show consistently significant improvement in the fit statistics-a finding of great use for researchers working predominantly with scales and constructs for their empirical models. However, based on the rudimentary experimental models employed in the analysis, attention checks generally do not show a consistent, systematic improvement in the significance of test statistics for experimental manipulations. This latter result indicates that, by their very nature, attention checks may trigger an inherent trade-off between loss of sample subjects-lowered power and increased Type II error-and the potential of capitalizing on chance alone-the possibility that the previously significant results were in fact the result of Type I error. The analysis also shows that the attrition rates due to attention checks-upwards of 70% in some observed samples-are far larger than typically assumed. Such loss rates raise the specter that studies not validating attention may inadvertently increase their Type I error rate. The manuscript provides general guidelines for various attention checks, discusses the psychological nuances of the methods, and highlights the delicate balance among incentive alignment, monetary compensation, and the subsequently triggered mood of respondents. (C) 2017 Elsevier B.V. All rights reserved.