The Generalizability of Online Experiments Conducted During the COVID-19 Pandemic
The Generalizability of Online Experiments Conducted During the COVID-19 Pandemic
复制标题
DOI：
10.1017/xps.2021.17
复制
发表时间：
2022-01-01
影响因子：
3.6
通讯作者：
Coppock, Alexander
中科院分区：
文献类型：
--
作者：
Peyton, Kyle;Huber, Gregory A.;Coppock, Alexander
COVID-19 大流行给实证研究带来了新的限制，社会科学家的在线数据收集也有所增加。由于参与者对治疗的反应或在线样本的构成发生了变化，从持续危机时期进行的实验中进行概括可能具有挑战性。我们对 2020 年 3 月至 7 月期间对 13 个美国人配额样本进行的 12 个大流行前设计进行了 33 次重复，研究了新冠病毒时代调查实验的普遍性。我们发现强有力的证据表明，大流行前的实验在迹象和意义方面进行了复制，但幅度有所降低。间接证据表明，在此期间，在线平台上注意力不集中的受试者比例有所增加，这可能导致估计治疗效果较小。总体而言，我们的结论是，大流行不会对在线实验在其他时期的推广造成根本威胁。
The COVID-19 pandemic imposed new constraints on empirical research, and online data collection by social scientists increased. Generalizing from experiments conducted during this period of persistent crisis may be challenging due to changes in how participants respond to treatments or the composition of online samples. We investigate the generalizability of COVID era survey experiments with 33 replications of 12 pre-pandemic designs, fielded across 13 quota samples of Americans between March and July 2020. We find strong evidence that pre-pandemic experiments replicate in terms of sign and significance, but at somewhat reduced magnitudes. Indirect evidence suggests an increased share of inattentive subjects on online platforms during this period, which may have contributed to smaller estimated treatment effects. Overall, we conclude that the pandemic does not pose a fundamental threat to the generalizability of online experiments to other time periods.