Empirical studies on the researcher degrees of freedom in biomedical data analysis

该项目从元科学的角度对作为卫生科学干预或观察性研究的一部分进行的统计分析进行了研究。因此，它处于元研究、统计学和医学的交界处。元科学的四个目标是(I)识别问题，(Ii)调查问题，(Iii)开发解决方案，(Iv)评估解决方案。我们的项目主要侧重于(二)和(三)，并可能包括(一)和(四)的组成部分。第一个项目阶段的重点是研究人员在统计分析中的自由度。虽然一些新的次级项目(WP1.1和1.2)将把第一阶段进行的研究扩展到提供解决办法，但也将探讨研究人员自由度的其他方面，即与生物医学研究设计有关的自由度(WP2.1和2.2)以及解释特定统计分析结果的自由度(WP3)。WP1.1的目标是在不依赖排列的多种分析策略的背景下，开发一种MINP类型的程序，用于多重测试调整。WP1.2的目标是对我们之前提出的以下非正式规则进行形式化、扩展和经验性的调查：“只有当作者在报告、讨论和解释他们的发现时，由于测试的重要性而更重视产生显著结果的测试时，才应该调整多重测试”。在WP2.1中，我们将对作为医学研究规划的一部分进行的影响样本量计算的研究人员自由度(不确定性)进行经验性调查并将其正式化，以提供试探性建议，而WP2.2的目的是研究所谓的适应性设计在临床试验范围之外的可行性，以解决因研究人员自由度而引起的不确定性。最后，在WP3中，我们将认为统计分析是固定的，并调查在解释和可视化阶段工作的自由度。

This project takes a metascientific perspective on statistical analyses performed as part of interventional or observational studies in health sciences. It is thus at the interface between meta-research, statistics and medicine. The four objectives of metascience are to (i) identify problems, (ii) investigate problems, (iii) develop solutions, and (iv) evaluate solutions. Our project focuses mainly on (ii) and (iii) and possibly includes components of (i) and (iv). The first project phase focused on the researcher degrees of freedom in statistical analyses. While some of the new subprojects (WP1.1 and 1.2) will extend the studies conducted in the first period towards providing solutions, further aspects of the researcher degrees of freedom will be explored as well, namely the degrees of freedom in relation to the design of biomedical studies (WP2.1 and 2.2) and the degrees of freedom in the interpretation of the results of given statistical analyses (WP3). The objective of WP1.1 is to develop a minP-type procedure for multiple testing adjustment in the context of the multiplicity of analysis strategies that does not rely on permutations. The objective of WP1.2 is to formalize, extend and empirically investigate the following informal rule we proposed previously: “Multiple testing should be adjusted for only in cases where authors put more weight on tests that yield significant results because of their significance when reporting, discussing, and interpreting their findings”. In WP2.1, we will empirically investigate and formalize the researcher degrees of freedom (uncertainties) that affect sample size calculations performed as part of the planning of medical studies, in order to provide tentative recommendations, while the aim of WP2.2 is to examine the feasibility of so-called adaptive designs beyond the context of clinical trials for addressing the uncertainty arising from the researcher degree of freedom. Finally, in WP3, we will consider the statistical analysis as fixed and investigate the degrees of freedom at work at the interpretation and visualization stage.