Methods for big data, sparsity, and environmental thresholds

随着技术的快速进步，科学和工程研究正在经历一场革命，因为大数据被用于各种目的。因此，该研究计划的长期目标是通过利用大数据中的有用信息来推进统计学，机器学习和统计生态学的科学。短期目标正在制定：基于不同变量子集的模型集合，以提高对药物发现和遗传学大数据应用的感兴趣响应变量的预测性能，贝叶斯统计测试和遗传学大数据中稀疏性的推断，以及生态模型，通过斜率和方差的变化来检测环境阈值，以对抗人类对自然的干扰。与丢弃信息不同，申请人提出了利用更多有用变量而不是丢弃一些变量的集成方法。假设是过滤变量就像丢失信息，这会降低模型的预测能力。为了解决这个问题，申请人提出了将有用的变量分组到不同的子集中并且通过将模型拟合到每个子集并且跨子集聚合模型来将它们聚合在集合中的方法。该方法的新奇在于自适应地获得变量的子集，而不是使用已知或随机子集的当前趋势。许多统计方法在建模时都强调稀疏性，即大数据中信息稀缺的情况，缺乏足够的理由和正当性。相反，申请人提出了在模型构建之前使用的稀疏性的贝叶斯统计测试。该方法的新奇之处在于，在模型构建中引入的稀疏度的量由数据中测量的证据支持。生态响应与环境干扰之间的关系往往通过斜率和方差的变化表现为阈值效应。目前，这种关系通过斜率或方差的变化由模型表示。有必要开发一种模型，可以通过斜率和方差的变化来估计阈值效应。此外，许多生态学家通常知道在压力-反应关系中斜率和方差的变化可能发生在哪里。申请人提出了一种贝叶斯模型，该模型可以通过添加来自生态学家的先验知识来同时估计斜率和方差的变化。该计划将通过更好地利用大数据中的信息来推进统计学研究。在药物发现和遗传学方面的应用将改善人类和动物的健康，并有可能开发新药和确定疾病的原因。在环境生态学方面，该研究将确定人类对自然造成的干扰的阈值效应，从而对水生生境进行可持续管理。经过培训的HQP将成为加拿大及其他地区学术界和工业界的资产。

With the rapid progress of technology, research in science and engineering is undergoing a revolution as big data are harnessed for various purposes. Thus, the long-term goal of this research program is to advance the science in statistics, machine learning and statistical ecology by exploiting the useful information in big data. The short-term goals are developing: ensemble of models based on diverse subsets of variables to improve prediction performances for the response variable of interest with applications to big data in drug discovery and genetics, Bayesian statistical tests and inference for sparsity in big data with applications to genetics, and ecological models to detect environmental thresholds via the changes in slope and variance against human induced disturbances to nature. Unlike discarding information, the applicant proposes ensemble methods which utilize more useful variables instead of throwing some out. The hypothesis is that filtering variables is like losing information which reduces the prediction power of a model. To address this issue, the applicant proposes methods which group useful variables into diverse subsets and aggregates them in an ensemble by fitting a model to each subset and aggregating models across the subsets. The novelty of the method is that the subsets of variables are obtained adaptively, rather than the current trend of using either known or random subsets. Many statistical methods force sparsity, the condition of scant information in big data, in model building which lacks enough reasons and justification. Instead, the applicant proposes Bayesian statistical tests for sparsity to be used before model building. The novelty of the method is that the amount of sparsity to induce in model building is supported by the evidence measured in data. Relationship between an ecological response and environmental disturbance often represent threshold effects via the changes in slope and variance. Presently, such relationship is represented by a model via the changes in either slope or variance. There is a need for developing a model which can estimate threshold effects via the changes in slope and variance. Also, many ecologists often know where the changes in slope and variance might occur in the stress-response relationship. The applicant proposes a Bayesian model which can estimate the changes in slope and variance, simultaneously, by adding the prior knowledge from ecologists. The proposed program will advance the research in statistics by better utilizing the information in big data. The applications in drug discovery and genetics will improve human and animal health with a potential to develop new drugs and identification of the causes of disease. In environmental ecology, the research will identify threshold effects of human induced disturbances to nature leading to sustainable management of aquatic habitat. The trained HQPs will be an asset in academia and industries in Canada and beyond.