A practical guide to understanding and validating complex models using data simulations
A practical guide to understanding and validating complex models using data simulations
复制标题
DOI：
10.1111/2041-210x.14030
复制
发表时间：
2022-11-18
影响因子：
6.6
通讯作者：
Miller, David A. W.
中科院分区：
文献类型：
--
作者：
DiRenzo, Graziella V.;Hanks, Ephraim;Miller, David A. W.
生物学家经常采用新颖而复杂的统计模型来突破我们理解的极限。示例包括但不限于灵活的贝叶斯方法（例如 BUGS、stan）、基于频率和可能性的方法（例如包 lme4）和机器学习方法。这些软件和程序为用户在定制复杂的分层模型方面提供了更大的控制和灵活性。然而，这种程度的控制和灵活性让用户承担了更高程度的责任来评估其统计推断的稳健性。为了确定生物学家对分层模型运行模型诊断的频率，我们回顾了 2021 年《自然生态与进化》杂志上最近发表的 50 篇论文，我们发现大多数已发表的论文没有报告其分层模型的任何验证，这使得读者很难评估其推论的稳健性。缺乏报告可能源于缺乏最佳实践和标准方法的标准化指导。在这里，我们提供了使用数据模拟理解和验证复杂模型的指南。为了确定生物学家使用数据模拟技术的频率，我们还回顾了 2021 年《方法生态学与进化》杂志上最近发表的 50 篇论文。我们发现，78% 提出新估计技术、包或模型的论文使用了模拟或以某种方式生成数据（23 篇论文中的 18 篇）；但这些论文中很少有（23 篇论文中的 5 篇）包括演示代码可以恢复对已知参数的数据集的实际估计，或演示该方法的统计特性。为了提炼各种模拟技术及其用途，我们根据预期的推论提供了模拟研究的分类法。我们还鼓励作者在使用新颖的统计模型时进行基本验证研究，这通常很容易实现。模拟数据有助于研究人员更深入地了解模型及其假设，并建立其估计方法的可靠性。生物学家更广泛地采用数据模拟可以改善统计推断、可靠性和开放科学实践。
Biologists routinely fit novel and complex statistical models to push the limits of our understanding. Examples include, but are not limited to, flexible Bayesian approaches (e.g. BUGS, stan), frequentist and likelihood-based approaches (e.g. packages lme4) and machine learning methods.These software and programs afford the user greater control and flexibility in tailoring complex hierarchical models. However, this level of control and flexibility places a higher degree of responsibility on the user to evaluate the robustness of their statistical inference. To determine how often biologists are running model diagnostics on hierarchical models, we reviewed 50 recently published papers in 2021 in the journal Nature Ecology & Evolution, and we found that the majority of published papers did not report any validation of their hierarchical models, making it difficult for the reader to assess the robustness of their inference. This lack of reporting likely stems from a lack of standardized guidance for best practices and standard methods.Here, we provide a guide to understanding and validating complex models using data simulations. To determine how often biologists use data simulation techniques, we also reviewed 50 recently published papers in 2021 in the journal Methods Ecology & Evolution. We found that 78% of the papers that proposed a new estimation technique, package or model used simulations or generated data in some capacity (18 of 23 papers); but very few of those papers (5 of 23 papers) included either a demonstration that the code could recover realistic estimates for a dataset with known parameters or a demonstration of the statistical properties of the approach. To distil the variety of simulations techniques and their uses, we provide a taxonomy of simulation studies based on the intended inference. We also encourage authors to include a basic validation study whenever novel statistical models are used, which in general, is easy to implement.Simulating data helps a researcher gain a deeper understanding of the models and their assumptions and establish the reliability of their estimation approaches. Wider adoption of data simulations by biologists can improve statistical inference, reliability and open science practices.