Scalable Blackbox Optimisation for Machine Learning

我的工作与EPSRC研究领域：信息和通信技术高度相关，因为我寻求解决影响人工智能研究前沿的问题。最近的工作表明，像进化策略(ES)这样的准随机搜索方法可以扩展到数百万个维度，在各种强化学习问题中取得了与最新技术相比具有竞争力的性能。这一令人兴奋的结果引发了一系列研究活动，并挑战了许多人认为可能的ES方法。尽管令人兴奋，ES方法仍然是令人难以置信的低效样本，因为它们基本上忽略了一次使用后的所有数据。因此，它们目前需要大量的并行计算资源才能取得令人印象深刻的结果。相反，贝叶斯优化方法的效率要高出许多个数量级，通常只需几次试验就能解决问题。贝叶斯优化已经成功地应用于许多工业环境，甚至导致了最近的AlphaGo算法的戏剧性改进，该算法击败了世界上最好的围棋选手--这是人工智能研究的一个里程碑式的结果。此外，贝叶斯优化方法可以使用原则性的数学方法来实现这些结果，而这是大多数高性能ES算法严重缺乏的。然而，大多数贝叶斯优化的成功案例来自较低维度的机制，它们通常被认为缺乏其他方法的可扩展性。尽管解决了同样的问题，但这两个社区仍然截然不同。我的目标是将它们结合在一起，建立样本高效的黑匣子优化算法，可以扩展到高维，并有理论支撑。我认为，这一领域存在重大机遇，但由于涉及到巨大的意识形态分歧，这些机遇尚未得到探索。到目前为止，异花授粉的情况很少，因为这两种方法处于光谱的两端。在我研究的第一年，我计划从几个不同的方向来解决这个问题：1)将贝叶斯方法引入最近引入的基于种群的训练算法，被证明显著提高了神经网络的性能。2)在优化期间使用多样性的概率测量来促进ES代理的种群的探索。3)使用贝叶斯原理在强化学习环境中主动获取数据。在第二年和第三年，我希望不仅引入更多新颖的算法，还希望在现实世界中展示这些方法的有效性。我相信这项研究可能会产生很大的影响，因为黑盒优化器被用于各种环境中，如强化学习，或工业规模深度学习项目的超参数调整。所有这些设置在下游都有多个用例，放大了影响。

My work is highly relevant for the EPSRC Research Area: Information and Communication Technologies, as I seek to solve problems impacting the forefront of Artificial Intelligence research. Recent work has shown that quasi-random search methods such as Evolution Strategies (ES) can scale to millions of dimensions, achieving competitive performance vs. state of the art in a variety of reinforcement learning problems. This exciting result has led to a flurry of research activity, and challenged what many thought was possible for ES methods. Despite this excitement, ES methods remain incredibly sample inefficient, since they essentially disregard all data after a single use. As such, they currently require large parallel computing resources to achieve their impressive results. Conversely, Bayesian Optimisation methods are many orders of magnitude more efficient, often solving problems in only a handful of trials. Bayesian Optimisation has been successfully used in many industrial settings, and even led to dramatic improvements in the recent AlphaGo algorithm which beat the world's best Go player - a landmark result for Artificial Intelligence research. Furthermore, Bayesian Optimisation methods can achieve these results using a principled mathematical approach, which is severely lacking in most high performing ES algorithms. However, most Bayesian Optimisation success stories come from a lower dimensional regime, and they are generally regarded to lack the scalability of other methods. Despite tackling the same problem, these two communities remain distinct. It is my goal to bring them together, to build sample efficient blackbox optimisation algorithms which can scale to high dimensions, with a theoretical underpinning. I believe there are significant opportunities in this area, which are yet to be explored due to the vast ideological differences involved. As of now, there has been very little cross-pollination, since the two methods are at opposite ends of the spectrum. In the first year of my studies, I plan to tackle the problem from several different directions:1) Bringing a Bayesian approach to the recently introduced Population Based Training algorithm, shown to dramatically improve performance for neural networks.2) Using probabilistic measures of diversity to boost exploration in populations of ES agents during optimisation.3) Using Bayesian principles to actively acquire data in a reinforcement learning setting. In the second and third years, I hope to not only introduce more novel algorithms, but also demonstrate the effectiveness of these approaches in real-world settings. I believe this research could have a large impact, since blackbox optimisers are used in a variety of settings such as reinforcement learning, or hyper-parameter tuning for industrial scale deep learning projects. All of these settings have multiple use cases downstream, magnifying the impact.