Methods for big data, sparsity, and environmental thresholds

随着技术的快速进步，科学和工程研究正在经历一场革命，因为大数据被用于各种目的。因此，这项研究计划的长期目标是通过利用大数据中的有用信息来推进统计学、机器学习和统计生态方面的科学。短期目标是：基于不同变量子集的模型集成，以改善感兴趣的响应变量的预测性能，将其应用于药物发现和遗传学中的大数据；将贝叶斯统计测试和推理用于大数据，将其应用于遗传学；以及生态模型，通过斜率和方差的变化来检测环境阈值，以对抗人类对自然的干扰。与丢弃信息不同，申请人提出的集成方法利用了更多有用的变量，而不是丢弃一些变量。其假设是，过滤变量就像丢失信息，这降低了模型的预测能力。为了解决这个问题，申请人提出了将有用的变量分组为不同子集并通过将模型适配到每个子集并跨子集聚集模型来将它们聚集在整体中的方法。该方法的创新之处在于，变量的子集是自适应获得的，而不是目前使用已知或随机子集的趋势。许多统计方法在建模时强制使用稀疏性，这是大数据中信息稀缺的情况，缺乏足够的理由和合理性。相反，申请人建议在建立模型之前使用稀疏性的贝叶斯统计测试。该方法的新奇之处在于，在模型建立中诱导的稀疏量得到了数据测量的证据的支持。生态响应与环境干扰之间的关系往往通过坡度和方差的变化表现为门槛效应。目前，这种关系是通过斜率或方差的变化用模型来表示的。需要开发一种模型，该模型可以通过斜率和方差的变化来估计阈值效应。此外，许多生态学家经常知道，在胁迫-响应关系中，坡度和方差的变化可能发生在哪里。申请人提出了一种贝叶斯模型，通过加入生态学家的先验知识，该模型可以同时估计坡度和方差的变化。拟议的计划将通过更好地利用大数据中的信息来推进统计研究。在药物发现和遗传学方面的应用将改善人类和动物的健康，有可能开发新药和确定疾病原因。在环境生态学方面，这项研究将确定人类对自然的干扰的阈值效应，从而实现对水生栖息地的可持续管理。经过培训的HQP将成为加拿大及其他地区学术界和工业界的一笔财富。

With the rapid progress of technology, research in science and engineering is undergoing a revolution as big data are harnessed for various purposes. Thus, the long-term goal of this research program is to advance the science in statistics, machine learning and statistical ecology by exploiting the useful information in big data. The short-term goals are developing: ensemble of models based on diverse subsets of variables to improve prediction performances for the response variable of interest with applications to big data in drug discovery and genetics, Bayesian statistical tests and inference for sparsity in big data with applications to genetics, and ecological models to detect environmental thresholds via the changes in slope and variance against human induced disturbances to nature. Unlike discarding information, the applicant proposes ensemble methods which utilize more useful variables instead of throwing some out. The hypothesis is that filtering variables is like losing information which reduces the prediction power of a model. To address this issue, the applicant proposes methods which group useful variables into diverse subsets and aggregates them in an ensemble by fitting a model to each subset and aggregating models across the subsets. The novelty of the method is that the subsets of variables are obtained adaptively, rather than the current trend of using either known or random subsets. Many statistical methods force sparsity, the condition of scant information in big data, in model building which lacks enough reasons and justification. Instead, the applicant proposes Bayesian statistical tests for sparsity to be used before model building. The novelty of the method is that the amount of sparsity to induce in model building is supported by the evidence measured in data. Relationship between an ecological response and environmental disturbance often represent threshold effects via the changes in slope and variance. Presently, such relationship is represented by a model via the changes in either slope or variance. There is a need for developing a model which can estimate threshold effects via the changes in slope and variance. Also, many ecologists often know where the changes in slope and variance might occur in the stress-response relationship. The applicant proposes a Bayesian model which can estimate the changes in slope and variance, simultaneously, by adding the prior knowledge from ecologists. The proposed program will advance the research in statistics by better utilizing the information in big data. The applications in drug discovery and genetics will improve human and animal health with a potential to develop new drugs and identification of the causes of disease. In environmental ecology, the research will identify threshold effects of human induced disturbances to nature leading to sustainable management of aquatic habitat. The trained HQPs will be an asset in academia and industries in Canada and beyond.