Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis
Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis
复制标题
DOI：
10.1016/j.obhdp.2021.02.003
复制
发表时间：
2021-07-13
影响因子：
4.6
通讯作者：
Uhlmann, Eric Luis
中科院分区：
文献类型：
--
作者：
Schweinsberg, Martin;Feldman, Michael;Uhlmann, Eric Luis
在这项众包计划中，独立分析师使用相同的数据集来测试关于科学家性别和专业地位对小组会议期间冗长程度影响的两个假设。不仅分析方法而且关键变量的操作都不受限制并由个别分析师决定。例如，分析师可以选择将状态具体化为职称、机构排名、引用计数或某种组合。为了最大限度地提高分析选择过程的透明度，分析师使用我们开发的名为 DataExplained 的平台来实时证明首选和拒绝的分析路径的合理性。缺乏足够细节、可重复代码或存在统计错误的分析被排除在外，最终样本中有 29 项分析。研究人员报告了截然不同的分析和分散的实证结果，在许多情况下，同一研究问题在相反的方向上获得了显着的效果。 Boba 多元宇宙分析表明，有关如何操作变量的决策可以解释超出统计选择（例如协变量）的结果的变异性。研究人员的主观决策在推动报告的实证结果方面发挥着关键作用，强调了开放数据、系统稳健性检查以及所采取和未采取的分析路径的透明度的必要性。讨论了对组织和领导者的影响，他们的决策部分依赖于科学发现、咨询报告和数据科学家的内部分析。
In this crowdsourced initiative, independent analysts used the same dataset to test two hypotheses regarding the effects of scientists' gender and professional status on verbosity during group meetings. Not only the analytic approach but also the operationalizations of key variables were left unconstrained and up to individual analysts. For instance, analysts could choose to operationalize status as job title, institutional ranking, citation counts, or some combination. To maximize transparency regarding the process by which analytic choices are made, the analysts used a platform we developed called DataExplained to justify both preferred and rejected analytic paths in real time. Analyses lacking sufficient detail, reproducible code, or with statistical errors were excluded, resulting in 29 analyses in the final sample. Researchers reported radically different analyses and dispersed empirical outcomes, in a number of cases obtaining significant effects in opposite directions for the same research question. A Boba multiverse analysis demonstrates that decisions about how to operationalize variables explain variability in outcomes above and beyond statistical choices (e.g., covariates). Subjective researcher decisions play a critical role in driving the reported empirical results, underscoring the need for open data, systematic robustness checks, and transparency regarding both analytic paths taken and not taken. Implications for orga-nizations and leaders, whose decision making relies in part on scientific findings, consulting reports, and internal analyses by data scientists, are discussed.