Statistical Problems Through a New Perturbation Theory

统计学和数据科学的主要目标之一是从大量数据中推断信息，这些数据通常以大矩阵的形式给出。例如，在大多数推荐系统中，矩阵的行表示客户，列表示产品，条目是客户对相应产品的（潜在）评级。由于数据带有噪声，因此测量噪声对我们想要推断的信息的影响是很重要的（在大多数情况下，根据一些矩阵参数）。鉴于光谱方法在数据科学中的普及，这个问题已经被许多领域的科学家研究了一个世纪。然而，在现代数据科学中，我们经常观察到，数据矩阵具有某种结构，例如低秩。PI的目标是在这种限制下发展一种新的理论，这将改进许多经典的数学结果，同时，将为许多现实生活中的问题带来更好的估计和更快的算法。该项目还为研究生提供研究培训机会。考虑一个矩阵（表示数据）。在实践中，我们经常使用它的噪声版本，其中每个条目都添加了一些噪声（或者将整个矩阵添加到噪声矩阵中）。经典的微扰定理，如Weyl或Davis-Kahan，为我们提供了原始矩阵关键参数（如前导特征值和特征向量）与其噪声对应参数之间差异的估计。这些结果在最坏的情况下分析是尖锐的。然而，在现代数据科学中，数据往往具有低秩结构，噪声是随机的。在这些假设下，PI观察到可以提供更好的边界。在这个项目中，在上述假设下，在各种范数下，PI将获得最优界。PI和他的学生还将使用结合这些新工具的谱方法来解决几个众所周知的算法问题，例如矩阵补全和高斯混合。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

One of the main goals of statistics and data science is to deduce information from a large set of data, usually given in the form of a large matrix. For instance, in most recommendation systems, the rows of the matrix represent customers, and the columns represent products, the entries are the (potential) rating of a customer for the corresponding product. Since data comes with noise, it is important to measure the impact of noise on the information we would like to deduce (in most cases, in terms of some matrix parameters). Given the popularity of the spectral method in data science, the problem has been studied by scientists across many fields for a century. However, in modern data science, it has been observed that very often, the data matrix has some structure, such as being low rank. The PI aims to develop a new theory with this restriction, which would improve many classical mathematical results, and at the same time, would lead to better estimates and faster algorithms for many real-life problems. The project also provides research training opportunities for graduate students. Consider a matrix (which represents data). In practice, we often have access to a noise version of it, where each entry is added with some noise (or the whole matrix is added to a noise matrix). Classical perturbation theorems, such as Weyl or Davis-Kahan, provide us with estimates for the difference between key parameters of the original matrix (such as leading eigenvalues and eigenvectors) and their noisy counterparts. These results are sharp in worst cases analysis. However, in modern data science, data often has a low-rank structure, and noise is random. Under these assumptions, the PI has observed that one can provide much better bounds. Under this project, the PI will obtain optimal bounds under the above assumption, in various norms. The PI and his students will also attack several well-known algorithmic problems, such as matrix completion and Gaussian mixtures, using the spectral method combined with these new tools.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.