Detecting computer-generated random responding in questionnaire-based data: A comparison of seven indices
Detecting computer-generated random responding in questionnaire-based data: A comparison of seven indices
复制标题
DOI：
10.3758/s13428-018-1103-y
复制
发表时间：
2019-10-01
影响因子：
5.4
通讯作者：
Cuneo, Felix
中科院分区：
文献类型：
--
作者：
Dupuis, Marc;Meier, Emanuele;Cuneo, Felix
随着在线数据收集和亚马逊的土耳其机器人（MTurk）等工具的发展，为了赚钱而生成调查响应的恶意软件的出现代表了一个主要问题，无论是出于经济还是科学原因。事实上，即使支付一个受访者完成一个问卷代表一个非常小的成本，僵尸网络提供无效的响应集的倍增可能最终降低研究的有效性，同时增加研究成本。到目前为止，已经提出了几种技术来检测有问题的人类反应集，但很少有研究已经进行了测试，他们实际上检测非人类反应集的程度。因此，我们建议对这些指数进行实证比较。假设大多数僵尸网络程序是基于随机均匀分布的响应，我们提出并比较了七个指标在这项研究中检测非人类的响应集。1,967名人类受访者的样本混合了不同的百分比（即，从5%到50%）的模拟随机响应集。七个指数中的三个（即，响应相干性、马氏距离和人-总相关性）似乎是检测非人类响应集的最佳估计量。考虑到其中的两个指标--马氏距离和人与总相关性--很容易计算，每个使用在线问卷的研究人员都可以用它们来筛选这些无效数据的存在。
With the development of online data collection and instruments such as Amazon's Mechanical Turk (MTurk), the appearance of malicious software that generates responses to surveys in order to earn money represents a major issue, for both economic and scientific reasons. Indeed, even if paying one respondent to complete one questionnaire represents a very small cost, the multiplication of botnets providing invalid response sets may ultimately reduce study validity while increasing research costs. Several techniques have been proposed thus far to detect problematic human response sets, but little research has been undertaken to test the extent to which they actually detect nonhuman response sets. Thus, we proposed to conduct an empirical comparison of these indices. Assuming that most botnet programs are based on random uniform distributions of responses, we present and compare seven indices in this study to detect nonhuman response sets. A sample of 1,967 human respondents was mixed with different percentages (i.e., from 5% to 50%) of simulated random response sets. Three of the seven indices (i.e., response coherence, Mahalanobis distance, and person-total correlation) appear to be the best estimators for detecting nonhuman response sets. Given that two of those indices-Mahalanobis distance and person-total correlation-are calculated easily, every researcher working with online questionnaires could use them to screen for the presence of such invalid data.