Statistical Problems Through a New Perturbation Theory

统计和数据科学的主要目标之一是从大量数据中推断信息，这些数据通常以大型矩阵的形式给出。例如，在大多数推荐系统中，矩阵的行表示客户，列表示产品，条目是客户对相应产品的(潜在)评级。由于数据带有噪声，因此重要的是测量噪声对我们想要推断的信息的影响(在大多数情况下，根据一些矩阵参数)。鉴于光谱方法在数据科学中的流行，这个问题已经被许多领域的科学家研究了一个世纪。然而，在现代数据科学中，人们已经观察到，数据矩阵往往具有某种结构，如低排名。PI旨在开发一种具有这种限制的新理论，它将改进许多经典数学结果，同时将导致许多实际问题的更好估计和更快的算法。该项目还为研究生提供了研究培训机会。考虑一个矩阵(表示数据)。在实践中，我们经常可以访问它的噪声版本，其中每个条目都添加了一些噪声(或者将整个矩阵添加到噪声矩阵中)。经典的扰动定理，如Weyl或Davis-Kahan，为我们提供了原始矩阵的关键参数(如引导特征值和特征向量)与其噪声对应的关键参数之间的差异的估计。在最糟糕的情况分析中，这些结果是尖锐的。然而，在现代数据科学中，数据往往具有低阶结构，噪声是随机的。在这些假设下，PI观察到一个人可以提供更好的界限。在这个项目下，PI将在上述假设下，在各种规范下获得最优界。PI和他的学生还将使用谱方法结合这些新工具来解决几个著名的算法问题，如矩阵完成和高斯混合。这个奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

One of the main goals of statistics and data science is to deduce information from a large set of data, usually given in the form of a large matrix. For instance, in most recommendation systems, the rows of the matrix represent customers, and the columns represent products, the entries are the (potential) rating of a customer for the corresponding product. Since data comes with noise, it is important to measure the impact of noise on the information we would like to deduce (in most cases, in terms of some matrix parameters). Given the popularity of the spectral method in data science, the problem has been studied by scientists across many fields for a century. However, in modern data science, it has been observed that very often, the data matrix has some structure, such as being low rank. The PI aims to develop a new theory with this restriction, which would improve many classical mathematical results, and at the same time, would lead to better estimates and faster algorithms for many real-life problems. The project also provides research training opportunities for graduate students. Consider a matrix (which represents data). In practice, we often have access to a noise version of it, where each entry is added with some noise (or the whole matrix is added to a noise matrix). Classical perturbation theorems, such as Weyl or Davis-Kahan, provide us with estimates for the difference between key parameters of the original matrix (such as leading eigenvalues and eigenvectors) and their noisy counterparts. These results are sharp in worst cases analysis. However, in modern data science, data often has a low-rank structure, and noise is random. Under these assumptions, the PI has observed that one can provide much better bounds. Under this project, the PI will obtain optimal bounds under the above assumption, in various norms. The PI and his students will also attack several well-known algorithmic problems, such as matrix completion and Gaussian mixtures, using the spectral method combined with these new tools.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.