Matrix Completion with Non-uniform Missing Patterns, a New Measure of Conditional Dependence, and Applications to Feature Selection

本课题旨在研究数理统计中的三类问题。第一类问题是关于矩阵补全的。假设我们有一个缺少条目的数字数组，比如一个产品用户评分的数据库。矩阵补全是预测缺失值的问题。在过去的十年里，人们对这个问题做了大量的研究，但绝大多数的研究都是在假设矩阵元素均匀随机丢失的情况下进行的。然而，在实践中，情况通常并非如此。本项目将实现一种方法，其中矩阵补全问题可以在更现实的假设下解决。这将影响矩阵补全算法应用的所有科学和技术领域，如推荐系统、协同过滤、计算机视觉和遗传学等。第二类问题涉及发展一种测量条件依赖性的方法。测量条件依赖性在统计学的许多应用中都很重要，例如在社会科学中广泛使用的图形模型和因果模型的分析中。第三类问题是关于开发一种新方法，用于在有大量变量时选择正确的变量进行回归分析。项目的这一部分将影响科学和技术的所有领域，在这些领域中，使用许多预测器的回归问题是常见的，例如生物学、医学和基因组学。矩阵补全课题旨在解决缺项模式确定时的低秩矩阵补全问题。PI最近发表了这个问题的渐近解。该项目将产生该理论的非渐近版本，以及当条目缺失的概率是条目本身的函数时矩阵补全的算法。关于条件依赖的新度量的项目将分析PI和他的一个学生最近提出的一个系数的渐近性质。分析的结果可能有助于设计新的条件独立性测试。特征选择项目将分析PI和他的一个学生最近提出的一种非参数特征选择算法的性质。分析结果可以更好地指导算法的实现，并产生新的和更好的选择算法。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This project aims to study three classes of problems in mathematical statistics. The first class of problems is about matrix completion. Suppose that we have an array of numbers with missing entries, such as a database of ratings from users of a product. Matrix completion is the problem of predicting the missing values. Much work has been done on this problem in the last ten years, but the vast majority of it is under the assumption that the matrix entries are missing uniformly at random. In practice, however, that is not usually the case. This project will implement a method where the matrix completion problem can be solved under more realistic assumptions. This will impact all areas of science and technology where matrix completion algorithms have applications, such as recommender systems, collaborative filtering, computer vision, and genetics, to name a few. The second class of problems concerns the development of an approach for measuring conditional dependence. Measuring conditional dependence is important in many applications of statistics, such as in the analysis of graphical and causal models, which are widely used in the social sciences. The third class of problems is about developing a new approach for selecting the right variables for performing regression analysis when presented with a large number of variables. This part of the project will impact all areas of science and technology where regression problems with many predictors are commonplace, such as biology, medicine, and genomics.The project on matrix completion aims to solve the low rank matrix completion problem when the pattern of missing entries is deterministic. The PI has recently published an asymptotic solution of the problem. The project will yield a non-asymptotic version of the theory, and an algorithm for matrix completion when the probability of an entry to be missing is a function of the entry itself. The project on a new measure of conditional dependence will analyze the asymptotic properties of a coefficient proposed recently by the PI and one of his students. The results of the analysis may help in devising new tests for conditional independence. The project on feature selection will analyze the properties of a non-parametric feature selection algorithm proposed recently by the PI and one of his students. The results of the analysis may guide better implementation of the algorithm, as well as yield new and better selection algorithms.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.