Tractable Big Data and Big Models in Machine Learning

点击翻译按钮获取中文摘要

In nearly all fields of science and engineering, the amount of data we collect is growing at unprecedented rates. We no longer produce data sizes from megabytes to gigabytes, but rather from terabytes to petabytes (and beyond). Machine learning is one of the key tools we use to make sense of these ever-growing quantities of data ('big data'), and it is now being used to solve very complicated tasks by fitting increasingly-complicated models to these huge data sets ('big models'). Machine learning is used in everyday technologies like e-mail spam filtering, product recommendation systems, advertisement ranking systems, new motion sensing devices, and recent improvements in speech recognition. There remains a huge potential for machine learning to impact applications ranging from physics to biology and from education technology to human-computer interaction. ***The successes and potential of large-scale machine learning are driving the need to develop techniques that can consider constantly-increasing data and model sizes, and the objective of this proposal is to advance the state of the art in fitting big models to big data sets. Building on my existing work showing that special model structures can be used to give order-of-magnitude improvements in runtimes,  the outcome of this research will be new techniques that are substantially faster than existing techniques (e.g., polynomial-time instead of exponential-time, or improving runtime by a factor that may be as large as the data or model size). In particular, the research will focus on:***1. Improved incremental gradient methods: this extends our recent work on exponentially-convergent stochastic gradient methods, leading to methods that have faster convergence rates, as well as memory-free methods that avoid expensive full passes through the data.***2. Exploiting new problem structures: this thread seeks out new problem structures to exploit (such as graphs and non-quadratic surrogate functions), which will lead to the discovery of new tractable problem classes.***3. Parallel and distributed methods: this thread focuses on developing methods that have appealing theoretical and practical properties when implemented in a parallel and distributed settings, allowing scaling to much larger datasets.***By focusing on improving the scalability of core enabling technologies behind data-driven applications, this work could affect a wide variety of scientific and engineering disciplines that produce huge datasets, in both academia and industry. Further, this research will provide key training to graduate and undergraduate students. It will produce highly-qualified personnel with skills and experience in large-scale data analysis, with the theoretical background required to solve ever-larger problems and model even more complex phenomena. These skills will prove to be a major asset in the expansion of Canada's growing knowledge-based economy.**