On robustness and adaptivity properties of mirror descent in high-dimension

最小化目标函数的问题自然出现在绝大多数统计学习问题中。有大量的优化方法来解决这个问题，其中许多都有很强的理论和经验保证。其中最简单和最知名的方法是梯度下降，沿着，其随机变量和广义镜像下降。然而，理论上通常研究的场景涉及未受污染的训练数据。考虑到如今学习算法必须处理的大量数据，人们会期望不可忽略的污染会对预测器的质量产生影响。在存在离群值的情况下的学习可以例如通过在学习阶段之前使用某种过滤方法来完成。这通常是计算密集型的，而且很难研究，因为不同类型的离群值需要不同的方法。如果优化算法与预处理步骤具有相同的效果，则可以节省大量的计算量，并采用统一的方法来处理此类问题。该项目的主要目标是了解镜像下降是否对“离群值”具有鲁棒性，以及是否可以适应数据中不同类型的污染。这个项目试图回答的问题如下：-是否有一个统一的方法来研究涉及不同类型的数据污染的统计学习问题？什么程度的污染可以证明导致任何学习算法具有较差的预测保证？镜像映射的选择（以及由此产生的几何形状）如何影响镜像下降算法的鲁棒性？是否有一个镜像映射的选择，以保证算法适应不同类型的数据污染？这些潜在的发现可以在多大程度上用于解释在不同模型（例如深度神经网络）中经验观察到的鲁棒性？自上个世纪由Huber开始的统计稳健性研究，主要集中在稳健估计方面，而对稳健预测的研究较少。在这一研究领域中使用的技术为解决上述问题提供了一个潜在的起点，沿着的还有关于高维统计、镜像下降和在线学习的文献（例如，在“两全其美”问题中）。我们希望通过结合上述领域中流行的不同方法，本项目的最终结果将是对鲁棒学习的一种新的理论理解，类似于经典学习理论框架。本项目福尔斯EPSRC统计和应用概率以及EPSRC理论计算机科学研究领域。

The problem of minimising an objective function naturally arises in the vast majority of statistical learning problems. There is a rich plethora of optimisation methods for solving this task, many of them with strong theoretical and empirical guarantees. One of the simplest and most well-known such methods is gradient descent, along with its stochastic variants and the generalized mirror descent. However, the scenarios commonly studied in theory involve non-contaminated training data. Given the enormous amounts of data learning algorithms have to process nowadays, one would expect non-negligible contaminations that would have an impact on the quality of the predictors. Learning in the presence of outliers can be done, for example, by using some filtration method prior to the learning phase. This often turns out to be computationally intensive and rather difficult to study, as different types of outliers require different approaches. If the optimisation algorithm would have the same effect as this pre-processing step, one might expect significant computational savings and a unified approach to dealing with such problems.The main goal of this project is to understand if mirror descent is robust to 'outliers', and if it possibly adapts to different types of contaminations in the data. The questions this project tries to answer are the following:- Is there a unified approach to studying statistical learning problems that involve different types of data contamination? What amount of contamination can provably cause any learning algorithm to have poor predictive guarantees?- How is the choice of the mirror map (and hence of the induced geometry) affecting the robustness properties of the mirror descent algorithm? Is there a choice of the mirror map that guarantees the algorithm is adaptive to different types of data contamination?- To what extent can these potential findings be applied in order to explain the robustness properties observed empirically in different models, e.g. deep neural networks?The work on statistical robustness, started in the previous century by Huber, mostly focuses on robust estimation, rather than robust prediction. Techniques used in this area of research offer a potential starting point to approach the above-mentioned questions, along with literature on high-dimensional statistics, mirror descent, and online learning (for example, as in the 'best of both worlds' problem). Our hope is that by combining different methods that are popular in the aforementioned areas, the final result of this project will be a novel theoretical understanding of robust learning, analogous to the classical learning theory framework.This project falls within the EPSRC Statistics and applied probability and EPSRC Theoretical computer science research areas.