Variability in the analysis of a single neuroimaging dataset by many teams.
Variability in the analysis of a single neuroimaging dataset by many teams.
复制标题
许多团队对单个神经影像数据集的分析存在差异
DOI：
10.1038/s41586-020-2314-9
复制
发表时间：
2020-06
期刊：
影响因子：
64.8
通讯作者：
Schonberg T
中科院分区：
文献类型：
--
作者：
Botvinik-Nezer R;Holzmeister F;Camerer CF;Dreber A;Huber J;Johannesson M;Kirchler M;Iwanir R;Mumford JA;Adcock RA;Avesani P;Baczkowski BM;Bajracharya A;Bakst L;Ball S;Barilari M;Bault N;Beaton D;Beitner J;Benoit RG;Berkers RMWJ;Bhanji JP;Biswal BB;Bobadilla-Suarez S;Bortolini T;Bottenhorn KL;Bowring A;Braem S;Brooks HR;Brudner EG;Calderon CB;Camilleri JA;Castrellon JJ;Cecchetti L;Cieslik EC;Cole ZJ;Collignon O;Cox RW;Cunningham WA;Czoschke S;Dadi K;Davis CP;Luca A;Delgado MR;Demetriou L;Dennison JB;Di X;Dickie EW;Dobryakova E;Donnat CL;Dukart J;Duncan NW;Durnez J;Eed A;Eickhoff SB;Erhart A;Fontanesi L;Fricke GM;Fu S;Galván A;Gau R;Genon S;Glatard T;Glerean E;Goeman JJ;Golowin SAE;González-García C;Gorgolewski KJ;Grady CL;Green MA;Guassi Moreira JF;Guest O;Hakimi S;Hamilton JP;Hancock R;Handjaras G;Harry BB;Hawco C;Herholz P;Herman G;Heunis S;Hoffstaedter F;Hogeveen J;Holmes S;Hu CP;Huettel SA;Hughes ME;Iacovella V;Iordan AD;Isager PM;Isik AI;Jahn A;Johnson MR;Johnstone T;Joseph MJE;Juliano AC;Kable JW;Kassinopoulos M;Koba C;Kong XZ;Koscik TR;Kucukboyaci NE;Kuhl BA;Kupek S;Laird AR;Lamm C;Langner R;Lauharatanahirun N;Lee H;Lee S;Leemans A;Leo A;Lesage E;Li F;Li MYC;Lim PC;Lintz EN;Liphardt SW;Losecaat Vermeer AB;Love BC;Mack ML;Malpica N;Marins T;Maumet C;McDonald K;McGuire JT;Melero H;Méndez Leal AS;Meyer B;Meyer KN;Mihai G;Mitsis GD;Moll J;Nielson DM;Nilsonne G;Notter MP;Olivetti E;Onicas AI;Papale P;Patil KR;Peelle JE;Pérez A;Pischedda D;Poline JB;Prystauka Y;Ray S;Reuter-Lorenz PA;Reynolds RC;Ricciardi E;Rieck JR;Rodriguez-Thompson AM;Romyn A;Salo T;Samanez-Larkin GR;Sanz-Morales E;Schlichting ML;Schultz DH;Shen Q;Sheridan MA;Silvers JA;Skagerlund K;Smith A;Smith DV;Sokol-Hessner P;Steinkamp SR;Tashjian SM;Thirion B;Thorp JN;Tinghög G;Tisdall L;Tompson SH;Toro-Serey C;Torre Tresols JJ;Tozzi L;Truong V;Turella L;van 't Veer AE;Verguts T;Vettel JM;Vijayarajah S;Vo K;Wall MB;Weeda WD;Weis S;White DJ;Wisniewski D;Xifra-Porxas A;Yearling EA;Yoon S;Yuan R;Yuen KSL;Zhang L;Zhang X;Zosky JE;Nichols TE;Poldrack RA;Schonberg T
许多科学领域的数据分析工作流程变得越来越复杂和灵活。在这里，我们通过要求70个独立的团队分析相同的数据集，测试相同的9个事前假设，评估这种灵活性对功能磁共振成像结果的影响。分析方法的灵活性体现在没有两个团队选择相同的工作流程来分析数据。这种灵活性导致假设检验的结果存在相当大的差异，即使对于统计图在分析流程的中间阶段高度相关的团队也是如此。报告结果的变化与分析方法的几个方面有关。值得注意的是，一种跨团队聚合信息的元分析方法在激活的区域中产生了重要的共识。此外，该领域研究人员的预测市场显示，即使是对数据集有直接了解的研究人员，也高估了重大发现的可能性。我们的研究结果表明，分析的灵活性可以对科学结论产生重大影响，并确定可能与功能磁共振成像分析中的变异性相关的因素。结果强调了验证和共享复杂分析工作流的重要性，并证明了对同一数据执行和报告多个分析的必要性。讨论了可用于缓解分析变异性相关问题的潜在方法。
Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses. The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset, , –. Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed.
登录
查看更多内容
影响因子：
5.7
作者：
LaConte, S;Anderson, J;Strother, S
通讯作者：
Strother, S
影响因子：
3.5
作者：
Abraham A;Pedregosa F;Eickenberg M;Gervais P;Mueller A;Kossaifi J;Gramfort A;Thirion B;Varoquaux G
通讯作者：
Varoquaux G
影响因子：
3.5
作者：
Gorgolewski KJ;Varoquaux G;Rivera G;Schwarz Y;Ghosh SS;Maumet C;Sochat VV;Nichols TE;Poldrack RA;Poline JB;Yarkoni T;Margulies DS
通讯作者：
Margulies DS
影响因子：
9.8
作者：
Botvinik-Nezer, Rotem;Iwanir, Roni;Schonberg, Tom
通讯作者：
Schonberg, Tom
影响因子：
1.6
作者：
Fountain, John;Harrison, Glenn W.
通讯作者：
Harrison, Glenn W.