Methods for big data, sparsity, and environmental thresholds

随着科技的快速进步，随着大数据被用于各种目的，科学和工程研究正在经历一场革命。因此，这个研究项目的长期目标是通过利用大数据中的有用信息来推进统计学、机器学习和统计生态学的科学发展。短期目标正在发展：基于不同变量子集的模型集成，以提高对感兴趣的响应变量的预测性能，应用于药物发现和遗传学的大数据；应用于遗传学的大数据中的贝叶斯统计检验和稀疏性推断；以及通过斜率和方差的变化来检测环境阈值的生态模型，以抵御人类对自然的干扰。与丢弃信息不同，申请人提出了使用更多有用变量而不是丢弃一些变量的集成方法。假设是，过滤变量就像丢失信息，这会降低模型的预测能力。为了解决这个问题，申请人提出了一些方法，这些方法将有用的变量分组到不同的子集中，并通过向每个子集拟合模型和跨子集聚集模型来将它们聚集在一个集合中。该方法的新颖之处在于变量的子集是自适应获得的，而不是目前使用已知或随机子集的趋势。许多统计方法在建立模型时强制使用稀疏性，这是大数据中缺乏信息的条件，缺乏足够的理由和证明。相反，申请人建议在模型构建之前使用贝叶斯统计测试来进行稀疏性测试。该方法的新颖之处在于，在模型构建中诱导的稀疏性的数量是由数据中测量的证据支持的。生态响应与环境扰动之间的关系通常表现为阈值效应，即斜率和方差的变化。目前，这种关系是通过斜率或方差的变化来表示的。有必要开发一种模型，可以通过斜率和方差的变化来估计阈值效应。此外，许多生态学家通常知道在应力-反应关系中斜率和方差的变化可能发生在哪里。申请人提出了一个贝叶斯模型，该模型可以同时估计斜率和方差的变化，通过添加生态学家的先验知识。该方案将通过更好地利用大数据信息来推进统计学研究。在药物发现和遗传学方面的应用将改善人类和动物的健康，并有可能开发新药和确定疾病的原因。在环境生态学方面，研究将确定人类对自然干扰的阈值效应，从而实现水生生境的可持续管理。训练有素的hqp将成为加拿大内外学术界和工业界的财富。

With the rapid progress of technology, research in science and engineering is undergoing a revolution as big data are harnessed for various purposes. Thus, the long-term goal of this research program is to advance the science in statistics, machine learning and statistical ecology by exploiting the useful information in big data. The short-term goals are developing: ensemble of models based on diverse subsets of variables to improve prediction performances for the response variable of interest with applications to big data in drug discovery and genetics, Bayesian statistical tests and inference for sparsity in big data with applications to genetics, and ecological models to detect environmental thresholds via the changes in slope and variance against human induced disturbances to nature. Unlike discarding information, the applicant proposes ensemble methods which utilize more useful variables instead of throwing some out. The hypothesis is that filtering variables is like losing information which reduces the prediction power of a model. To address this issue, the applicant proposes methods which group useful variables into diverse subsets and aggregates them in an ensemble by fitting a model to each subset and aggregating models across the subsets. The novelty of the method is that the subsets of variables are obtained adaptively, rather than the current trend of using either known or random subsets. Many statistical methods force sparsity, the condition of scant information in big data, in model building which lacks enough reasons and justification. Instead, the applicant proposes Bayesian statistical tests for sparsity to be used before model building. The novelty of the method is that the amount of sparsity to induce in model building is supported by the evidence measured in data. Relationship between an ecological response and environmental disturbance often represent threshold effects via the changes in slope and variance. Presently, such relationship is represented by a model via the changes in either slope or variance. There is a need for developing a model which can estimate threshold effects via the changes in slope and variance. Also, many ecologists often know where the changes in slope and variance might occur in the stress-response relationship. The applicant proposes a Bayesian model which can estimate the changes in slope and variance, simultaneously, by adding the prior knowledge from ecologists. The proposed program will advance the research in statistics by better utilizing the information in big data. The applications in drug discovery and genetics will improve human and animal health with a potential to develop new drugs and identification of the causes of disease. In environmental ecology, the research will identify threshold effects of human induced disturbances to nature leading to sustainable management of aquatic habitat. The trained HQPs will be an asset in academia and industries in Canada and beyond.