Scalable Online Machine Learning

点击翻译按钮获取中文摘要

Many of the current problems within the modern machine learning sector involve dealing with ever growing datasets where accurately and efficiently estimating a posterior distribution to make predictions is very difficult due to being able to identify the correct information from datasets which is informative to modelling our target distribution. As well as volume, these models often increase in dimensionality (and therefore complexity) which current common methods such as Markov Chain Monte Carlo (MCMC)- sampling struggle to do efficiently, as they become much slower as the complexity of the model increases. Other types of sampling methods though have shown to scale much better than traditional MCMC. For example, Sequential Monte Carlo sampling (SMC) and Hamiltonian Monte Carlo (HMC) sampling scale a lot better with dimensionality. These techniques are nowhere near as well researched though and have their own problems, so improving upon these further by introducing Reversible Jump (RJ) MCMC and Mass Matrices (MM) will be a main stay of my research. My colleague Josh Murphy will be working on Concept Drift, and I will be working on Eternal Learning. Initially there may be some similarities on the research we undertake but due to the differences in problem concepts, this will start to diverge within the first year. However, further down the line there will likely be some collaboration as the methods developed in each area may have some overlap depending on the specific problem we are undertaking. Therefore, I will also be researching methods to compress data from a constant and endless source but so that little to no information is lost during our inference with the aforementioned sampling methods. This will need to be done, as for eternal learning, the first sample will be just as important as the most recent one. Finally, these new methods will be applied to a Bayesian deep learning context. In neural networks, backpropagation for use in calculating the weights and biases of models is very computationally expensive. Early research with using particle filters as an alternative (or in an ensemble method) to backpropagation has recently started in the past couple of years as they are less computationally expensive. I will expand upon this by implementing a quasi-differentiable SMC sampler (as opposed to a particle filter) to aid the optimization process in neural networks.