Statistical Problems Through a New Perturbation Theory

统计学和数据科学的主要目标之一是从大量数据中推导出信息，这些数据通常以大型矩阵的形式给出。例如，在大多数推荐系统中，矩阵的行代表客户，列代表产品，条目是客户对相应产品的（潜在）评级。由于数据带有噪声，因此测量噪声对我们想要推断的信息的影响（在大多数情况下，根据一些矩阵参数）非常重要。鉴于谱方法在数据科学中的流行，这个问题已经被许多领域的科学家研究了世纪。然而，在现代数据科学中，已经观察到数据矩阵通常具有某种结构，例如低秩。PI的目标是开发一种具有这种限制的新理论，这将改善许多经典的数学结果，同时，将为许多现实问题带来更好的估计和更快的算法。该项目还为研究生提供研究培训机会。考虑一个矩阵（表示数据）。在实践中，我们经常可以访问它的噪声版本，其中每个条目都添加了一些噪声（或者整个矩阵都添加到噪声矩阵中）。经典的扰动定理，如Weyl或Davis-Kahan，为我们提供了原始矩阵的关键参数（如前导特征值和特征向量）与其噪声对应项之间差异的估计。这些结果在最差情况分析中非常明显。然而，在现代数据科学中，数据往往具有低秩结构，噪声是随机的。在这些假设下，PI观察到可以提供更好的边界。在这个项目下，PI将在上述假设下，在各种规范下获得最佳界限。PI和他的学生还将使用光谱方法结合这些新工具解决几个著名的算法问题，如矩阵完成和高斯混合。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估而被认为值得支持。

One of the main goals of statistics and data science is to deduce information from a large set of data, usually given in the form of a large matrix. For instance, in most recommendation systems, the rows of the matrix represent customers, and the columns represent products, the entries are the (potential) rating of a customer for the corresponding product. Since data comes with noise, it is important to measure the impact of noise on the information we would like to deduce (in most cases, in terms of some matrix parameters). Given the popularity of the spectral method in data science, the problem has been studied by scientists across many fields for a century. However, in modern data science, it has been observed that very often, the data matrix has some structure, such as being low rank. The PI aims to develop a new theory with this restriction, which would improve many classical mathematical results, and at the same time, would lead to better estimates and faster algorithms for many real-life problems. The project also provides research training opportunities for graduate students. Consider a matrix (which represents data). In practice, we often have access to a noise version of it, where each entry is added with some noise (or the whole matrix is added to a noise matrix). Classical perturbation theorems, such as Weyl or Davis-Kahan, provide us with estimates for the difference between key parameters of the original matrix (such as leading eigenvalues and eigenvectors) and their noisy counterparts. These results are sharp in worst cases analysis. However, in modern data science, data often has a low-rank structure, and noise is random. Under these assumptions, the PI has observed that one can provide much better bounds. Under this project, the PI will obtain optimal bounds under the above assumption, in various norms. The PI and his students will also attack several well-known algorithmic problems, such as matrix completion and Gaussian mixtures, using the spectral method combined with these new tools.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.