Towards neutral comparison studies in methodological biostatistics research

统计学家往往热衷于分析所谓的“复制危机”的统计方面。他们谴责利用统计方法在实证科学领域进行调查和发表偏见。但是，在他们自己的方法论研究中，即把统计方法作为研究对象的研究中，好的实践问题又如何呢？在开发和评估新的统计方法和数据分析工具时，统计学家是否遵守他们在应用统计的领域所提倡的良好实践原则？我认为，在统计学和数据科学的方法学研究背景下，统计学家应该做出实质性的努力来解决所谓的复制危机。第一个项目的目标是提高对与这种重复危机有关的问题的认识，重点是影响许多方法学研究的偏见。这将特别通过精心设计的经验研究来实现，这些研究也有助于确定改进战略。在此背景下，我们认为，中立的比较研究调查（生物）统计方法，经过精心设计，在评估方法时避免各种偏差的策略和严格的无偏倚报告，对于提高方法学文献的可靠性和允许更多基于证据和无偏倚的统计/数据分析方法的选择起着重要作用。该项目的第二个主要目标是开发、演示和评估尽可能“中立”地设计、实施、解释和报告比较研究的潜在策略，因此对中立概念的精确定义也将是该项目的一部分。该项目由三个相互关联的子项目组成，涉及比较研究的中立性：“比较研究和偏见的设计”、“中立比较研究的报告”和“处理比较研究的灵活性”。它的独创性在于它采用了一种元科学的方法来解决统计学和相关领域的方法问题，这是不寻常的。元研究尚未被方法论科学界视为一个成熟的研究领域（相反，在过去二十年中，在心理学和健康研究等其他领域已经观察到鼓舞人心的发展）。因此，该项目还旨在在统计学和数据科学的方法学研究背景下建立元科学，从长远来看，这不仅会提高该领域的研究质量，而且会提高应用统计/数据分析方法的所有领域的研究质量。

Statisticians are often keen to analyze the statistical aspects of the so-called “replication crisis”. They condemn fishing expeditions and publication bias across empirical scientific fields applying statistical methods. But what about good practice issues in their own - methodological - research, i.e. research considering statistical methods as research objects? When developing and evaluating new statistical methods and data analysis tools, do statisticians adhere to the good practice principles they promote in fields which apply statistics? I argue that statisticians should make substantial efforts to address what may be called the replication crisis in the context of methodological research in statistics and data science. The first project aim is to increase awareness of issues related to this replication crisis with a focus on biases affecting many methodological studies. This will be achieved in particular through well-designed empirical studies that also help to identify strategies for improvement. In this context, we argue that neutral comparison studies investigating (bio)statistical methods with thoughtfully elaborated design, strategies to avoid various types of bias when evaluating methods and rigorous unbiased reporting play an important role for increasing the reliability of methodological literature and allowing more evidence-based and unbiased choice of statistical/data analysis methods. The second – major - aim of the project is to develop, demonstrate and evaluate potential strategies to design, conduct, interpret and report comparison studies as “neutrally” as possible, whereby the precise definition of the concept of neutrality will also be part of the project. The project is structured in three interrelated subprojects dealing with the neutrality of comparison studies: “design of comparison studies and biases”, “reporting of neutral comparison studies”, and “handling of the flexibility of comparison studies”. Its originality is that it adopts a metascientific approach to address methodological issues in statistics and related fields, where it is unusual to do so. Meta-research is not yet perceived by the methodological scientific community as a full-fledged research area (in contrast, inspiring developments have been observed in other fields such as psychology and health research in the two last decades). The project thus also aims at establishing metascience in the context of methodological research on statistics and data science, which should, in the long run, lead to increased research quality not only in this field but also across all fields that apply statistical/data analysis methods.