Methods for big data, sparsity, and environmental thresholds

点击翻译按钮获取中文摘要

With the rapid progress of technology, research in science and engineering is undergoing a revolution as big data are harnessed for various purposes. Thus, the long-term goal of this research program is to advance the science in statistics, machine learning and statistical ecology by exploiting the useful information in big data. The short-term goals are developing: ensemble of models based on diverse subsets of variables to improve prediction performances for the response variable of interest with applications to big data in drug discovery and genetics, Bayesian statistical tests and inference for sparsity in big data with applications to genetics, and ecological models to detect environmental thresholds via the changes in slope and variance against human induced disturbances to nature. Unlike discarding information, the applicant proposes ensemble methods which utilize more useful variables instead of throwing some out. The hypothesis is that filtering variables is like losing information which reduces the prediction power of a model. To address this issue, the applicant proposes methods which group useful variables into diverse subsets and aggregates them in an ensemble by fitting a model to each subset and aggregating models across the subsets. The novelty of the method is that the subsets of variables are obtained adaptively, rather than the current trend of using either known or random subsets. Many statistical methods force sparsity, the condition of scant information in big data, in model building which lacks enough reasons and justification. Instead, the applicant proposes Bayesian statistical tests for sparsity to be used before model building. The novelty of the method is that the amount of sparsity to induce in model building is supported by the evidence measured in data. Relationship between an ecological response and environmental disturbance often represent threshold effects via the changes in slope and variance. Presently, such relationship is represented by a model via the changes in either slope or variance. There is a need for developing a model which can estimate threshold effects via the changes in slope and variance. Also, many ecologists often know where the changes in slope and variance might occur in the stress-response relationship. The applicant proposes a Bayesian model which can estimate the changes in slope and variance, simultaneously, by adding the prior knowledge from ecologists. The proposed program will advance the research in statistics by better utilizing the information in big data. The applications in drug discovery and genetics will improve human and animal health with a potential to develop new drugs and identification of the causes of disease. In environmental ecology, the research will identify threshold effects of human induced disturbances to nature leading to sustainable management of aquatic habitat. The trained HQPs will be an asset in academia and industries in Canada and beyond.