An MTurk Crisis? Shifts in Data Quality and the Impact on Study Results
An MTurk Crisis? Shifts in Data Quality and the Impact on Study Results
复制标题
DOI：
10.1177/1948550619875149
复制
发表时间：
2020-05-01
影响因子：
5.7
通讯作者：
Kucker, Sarah C.
中科院分区：
文献类型：
--
作者：
Chmielewski, Michael;Kucker, Sarah C.
亚马逊的机械土耳其(MTurk)可以说是过去十年中最重要的研究工具之一。快速收集大量高质量人类受试者数据的能力推动了包括人格和社会心理学在内的多个领域的发展。从2018年夏天开始，人们对MTurk的数据质量产生了担忧，导致人们对MTurk用于心理学研究的效用产生了质疑。我们使用四波自然主义实验设计提供了数据质量大幅下降的经验证据：2018年夏季之前、期间和之后。在2018年夏季期间和在一定程度上，我们发现参与者未能通过反应效度指标的人数显著增加，广泛使用的人格测量的信度和效度下降，以及未能复制既定的研究结果。然而，这些有害的影响可以通过使用响应有效性指标和筛选数据来减轻。我们讨论影响并提供建议以确保数据质量。
Amazon's Mechanical Turk (MTurk) is arguably one of the most important research tools of the past decade. The ability to rapidly collect large amounts of high-quality human subjects data has advanced multiple fields, including personality and social psychology. Beginning in summer 2018, concerns arose regarding MTurk data quality leading to questions about the utility of MTurk for psychological research. We present empirical evidence of a substantial decrease in data quality using a four-wave naturalistic experimental design: pre-, during, and post-summer 2018. During and to some extent post-summer 2018, we find significant increases in participants failing response validity indicators, decreases in reliability and validity of a widely used personality measure, and failures to replicate well-established findings. However, these detrimental effects can be mitigated by using response validity indicators and screening the data. We discuss implications and offer suggestions to ensure data quality.