Resampling-based comparison studies of prediction methods with emphasis on high-dimensional biological data

基于重采样的预测方法比较研究，重点关注高维生物数据

• 批准号: 158005760
• 负责人: Professorin Dr. Anne-Laure Boulesteix
• 金额: --
• 依托单位: Institut für Medizinische Informationsverarbeitung, Biometrie und Epidemiologie
• 依托单位国家: 德国
• 项目类别: Research Grants
• 财政年份: 2009
• 资助国家: 德国
• 起止时间: 2008-12-31 至 2017-12-31
• 项目状态: 已结题
• 来源: https://gepris.dfg.de/gepris/projekt/158005760?language=en
• 关键词: Resampling; based; comparison; studies; prediction

Resampling-based comparison studies of prediction methods using real data sets are routinely conducted in computational research, including, for instance, that for biostatistics, bioinformatics, or machine learning. For example, supervised classification methods may be compared with respect to their cross-validation error on, say, five exemplary real data sets. However, many methodological issues related to such comparison studies, and resampling-based methods in general, remain unanswered. This project deals with such issues with emphasis on applications to high-dimensional biological data, among others. In the first part of the project we address the design of comparison studies from a statistical testing perspective by drawing parallels between comparison studies in computational sciences (in which the performances of the methods are compared in real data sets) and clinical trials (in which therapy effects are compared in patients). In light of this metaphor we develop methods to address issues such as the choice of the resampling procedure with respect to its variability and the relationship between the performance of a method and the characteristics of the datasets, with emphasis on statistical inference and power. We extend this statistical framework to issues other than prediction models. The second part deals with scientific practice and the interpretation of literature in the field of computational science with focus on prediction methods while borrowing concepts from biomedical/clinical research. More precisely, we extend concepts such as meta-analysis, inclusion criteria for patients in clinical studies, and the degree of freedom of the researcher (related to multiple testing issues and fishing for significance) to the world of computational research. The goal of this part is to suggest first concepts and to investigate their feasibility. In the third part we develop methods related to parameter tuning for prediction methods, which is often performed using resampling methods in practice. We propose methods to measure the impact of tuning parameters on the performance of a method, we systematically assess and compare various resampling-based procedures for tuning, and we develop alternatives to resampling.

基于重采样的预测方法的比较研究，使用真实的数据集通常在计算研究中进行，包括，例如，生物统计学，生物信息学，或机器学习。例如，监督分类方法可以相对于它们在例如五个示例性真实的数据集上的交叉验证误差进行比较。然而，与这种比较研究有关的许多方法学问题，以及一般的基于重采样的方法，仍然没有答案。该项目处理这些问题，重点是高维生物数据的应用等。在该项目的第一部分，我们从统计检验的角度处理比较研究的设计，通过绘制计算科学（在真实的数据集中比较方法的性能）和临床试验（在患者中比较治疗效果）之间的比较研究之间的相似之处。根据这个比喻，我们开发的方法来解决的问题，如选择的restaurant程序相对于其可变性和方法的性能和数据集的特性之间的关系，重点是统计推断和权力。我们将这个统计框架扩展到预测模型以外的问题。第二部分涉及计算科学领域的科学实践和文献解释，重点是预测方法，同时从生物医学/临床研究中借用概念。更确切地说，我们扩展的概念，如荟萃分析，纳入标准的患者在临床研究中，研究人员的自由度（涉及多个测试问题和捕鱼的意义）的计算研究的世界。这一部分的目的是提出第一个概念，并调查其可行性。在第三部分中，我们开发了与预测方法的参数整定相关的方法，这在实践中通常使用reservation方法进行。我们提出的方法来衡量调整参数的方法的性能的影响，我们系统地评估和比较各种重采样为基础的程序调整，我们开发的替代方案，重新采样。