Matrix Completion with Non-uniform Missing Patterns, a New Measure of Conditional Dependence, and Applications to Feature Selection

本课题主要研究数理统计中的三类问题。第一类问题是关于矩阵完备化的。假设我们有一个缺少条目的数字数组，比如一个产品用户评分的数据库。矩阵完备化是预测缺失值的问题。在过去的十年里，人们对这个问题做了很多工作，但绝大多数都是在假设矩阵元素均匀随机缺失的情况下进行的。但实际上，情况通常并非如此。这个项目将实现一种方法，可以在更现实的假设下解决矩阵完成问题。这将影响矩阵补全算法应用的所有科学和技术领域，例如推荐系统，协同过滤，计算机视觉和遗传学等。第二类问题涉及到发展一种测量条件依赖的方法。测量条件依赖性在统计学的许多应用中很重要，例如在社会科学中广泛使用的图形和因果模型的分析中。第三类问题是关于开发一种新的方法来选择正确的变量进行回归分析时，提出了大量的变量。该项目的这一部分将影响所有科学和技术领域，其中许多预测因子的回归问题是司空见惯的，如生物学，医学和基因组学。矩阵补全项目旨在解决低秩矩阵补全问题时，丢失的条目的模式是确定的。PI最近发表了这个问题的渐近解。该项目将产生一个非渐近版本的理论，和一个算法的矩阵完成时，一个条目的概率是一个函数的条目本身。这个关于条件依赖的新度量的项目将分析PI和他的一个学生最近提出的一个系数的渐近性质。分析的结果可能有助于设计新的条件独立性测试。关于特征选择的项目将分析PI和他的一个学生最近提出的非参数特征选择算法的特性。分析的结果可以指导算法的更好的实现，以及产生新的和更好的选择algorithm.This奖项反映了NSF的法定使命，并已被认为是值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估的支持。

This project aims to study three classes of problems in mathematical statistics. The first class of problems is about matrix completion. Suppose that we have an array of numbers with missing entries, such as a database of ratings from users of a product. Matrix completion is the problem of predicting the missing values. Much work has been done on this problem in the last ten years, but the vast majority of it is under the assumption that the matrix entries are missing uniformly at random. In practice, however, that is not usually the case. This project will implement a method where the matrix completion problem can be solved under more realistic assumptions. This will impact all areas of science and technology where matrix completion algorithms have applications, such as recommender systems, collaborative filtering, computer vision, and genetics, to name a few. The second class of problems concerns the development of an approach for measuring conditional dependence. Measuring conditional dependence is important in many applications of statistics, such as in the analysis of graphical and causal models, which are widely used in the social sciences. The third class of problems is about developing a new approach for selecting the right variables for performing regression analysis when presented with a large number of variables. This part of the project will impact all areas of science and technology where regression problems with many predictors are commonplace, such as biology, medicine, and genomics.The project on matrix completion aims to solve the low rank matrix completion problem when the pattern of missing entries is deterministic. The PI has recently published an asymptotic solution of the problem. The project will yield a non-asymptotic version of the theory, and an algorithm for matrix completion when the probability of an entry to be missing is a function of the entry itself. The project on a new measure of conditional dependence will analyze the asymptotic properties of a coefficient proposed recently by the PI and one of his students. The results of the analysis may help in devising new tests for conditional independence. The project on feature selection will analyze the properties of a non-parametric feature selection algorithm proposed recently by the PI and one of his students. The results of the analysis may guide better implementation of the algorithm, as well as yield new and better selection algorithms.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.