Resampling-based comparison studies of prediction methods with emphasis on high-dimensional biological data

基于重采样的使用真实数据集的预测方法的比较研究通常在计算研究中进行，例如，包括用于生物统计学、生物信息学或机器学习的研究。例如，可以将监督分类方法相对于它们在例如五个示例性真实数据集上的交叉验证误差进行比较。然而，与这类比较研究有关的许多方法学问题，以及一般基于重新抽样的方法，仍然没有得到回答。这个项目处理这些问题，重点是高维生物数据的应用等。在项目的第一部分，我们从统计测试的角度解决比较研究的设计，通过比较计算科学中的比较研究(在真实数据集中比较方法的性能)和临床试验(在患者中比较治疗效果)来进行比较研究。根据这个比喻，我们开发了一些方法来解决一些问题，例如关于重采样过程的可变性的选择，以及方法的性能和数据集的特征之间的关系，重点是统计推断和能力。我们将这一统计框架扩展到预测模型以外的问题。第二部分涉及计算科学领域的科学实践和文献解释，重点是预测方法，同时借用生物医学/临床研究的概念。更准确地说，我们将Meta分析、临床研究中患者的纳入标准以及研究人员的自由度(与多个测试问题和寻找意义有关)等概念扩展到计算研究的世界。这一部分的目的是提出最初的概念并调查其可行性。在第三部分中，我们开发了与预测方法参数调整相关的方法，在实践中经常使用重采样方法来执行。我们提出了衡量调整参数对方法性能影响的方法，我们系统地评估和比较了各种基于重采样的调整过程，并开发了重采样的替代方法。

Resampling-based comparison studies of prediction methods using real data sets are routinely conducted in computational research, including, for instance, that for biostatistics, bioinformatics, or machine learning. For example, supervised classification methods may be compared with respect to their cross-validation error on, say, five exemplary real data sets. However, many methodological issues related to such comparison studies, and resampling-based methods in general, remain unanswered. This project deals with such issues with emphasis on applications to high-dimensional biological data, among others. In the first part of the project we address the design of comparison studies from a statistical testing perspective by drawing parallels between comparison studies in computational sciences (in which the performances of the methods are compared in real data sets) and clinical trials (in which therapy effects are compared in patients). In light of this metaphor we develop methods to address issues such as the choice of the resampling procedure with respect to its variability and the relationship between the performance of a method and the characteristics of the datasets, with emphasis on statistical inference and power. We extend this statistical framework to issues other than prediction models. The second part deals with scientific practice and the interpretation of literature in the field of computational science with focus on prediction methods while borrowing concepts from biomedical/clinical research. More precisely, we extend concepts such as meta-analysis, inclusion criteria for patients in clinical studies, and the degree of freedom of the researcher (related to multiple testing issues and fishing for significance) to the world of computational research. The goal of this part is to suggest first concepts and to investigate their feasibility. In the third part we develop methods related to parameter tuning for prediction methods, which is often performed using resampling methods in practice. We propose methods to measure the impact of tuning parameters on the performance of a method, we systematically assess and compare various resampling-based procedures for tuning, and we develop alternatives to resampling.