Random Matrices in Multivariate Statistics: Theoretical Developments and Applications

这个研究项目目前的重点是为高维问题的新范式开发数据分析方法。相关的理论问题涉及到大维随机矩阵的特征值问题。更确切地说，三个相关的方向似乎特别感兴趣：1)进一步我们对相关随机矩阵的谱性质的理解；2)结合随机矩阵理论的一些较为经典的结果，对所得结果进行实际应用；3)找到与此框架相关的应用领域并做出贡献。更具体地说，现在统计学家经常面临“n乘以p”的数据矩阵X，其中p与n具有相同的数量级，并且p和n都很大。从这些数据中计算出的样本协方差矩阵对许多应用都非常重要，因为它是主成分分析等广泛使用的方法的基础。然而，该方法的理论结果不适用于刚刚描述的“大n，大p”设置。因此，在这种情况下，需要对样本协方差矩阵进行深入的研究。这种大维矩阵的特征值是特别有趣的。从应用的角度来看，这些矩阵的最大和最小特征值是特别有趣的。本研究的目的是获得这些极端特征值的中心极限型定理，并将其用于统计学中，例如，假设检验，幂的概念等。这项工作的一个更实用的部分是有效地利用随机矩阵理论的结果-新的和旧的-更好地估计总体协方差的特征值，最终目的是在p和n都很大时更好地估计整个协方差矩阵。技术进步使我们能够存储和使用关于日常生活许多方面的大量数据。一个有趣的问题是使用这些数据来理解某些特征是如何相互依赖的。在股票市场中，我们可能感兴趣的是一只股票的行为如何影响另一只股票的行为；了解了所有这些相互关系，就可以对使用相应股票的投资组合所承担的风险进行衡量。统计学家有许多工具来处理所有这些相互关系。我们可以找到查看数据的方法，即使所有的相互关系都很小或很弱，所以每个特征“不应该”帮助我们了解太多其他特征，我们可能会发现携带大量信息的特征组合。我们还知道这些组合的典型值是什么，因此我们可以通过正确的方式检测数据中的异常情况。这些统计技术在各个科学领域有着非常广泛的应用，从气候学到遗传学，图像识别等。每年发表的数千篇研究论文都使用了这些技术。然而，这些统计技术背后的理论是在一个大规模数据集还不存在的时代创建的，因为它们是不可存储的。这个研究项目的重点是理论及其应用，这些理论和应用更适合处理我们当前的海量数据集。应用程序应该允许我们看到经典工具看不到的结构，并且在经典工具告诉我们有结构的时候告诉我们什么时候没有结构。我们也有越来越多的证据表明，我们的标准工具给我们的结果通常是非常不准确的，关于我们的风险标准测量或特征组合中携带的信息量。似乎风险可能被低估了，信息量可能被高估了。该研究计划的一部分将致力于测量大型数据集的经典结果有多不准确，以及如何使用更相关的理论来纠正这些不准确性。

This research program is currently focused on the development of data analysis methods for the new paradigm of high-dimensional problems. The associated theoretical problems are concerned with eigenvalues of large dimensional random matrices. More precisely, three related directions seem of particular interest: 1) further our understanding of the spectral properties of the relevant random matrices; 2) make practical use of the results obtained, combined with some more classical results from random matrix theory; 3) find and contribute to area of applications where this framework is relevant. More specifically, it is now very often that statisticians are faced with ``n times p" data matrices X, for which p, is of the same order of magnitude as n, and p and n are both large. The sample covariance matrix computed from this data is of great importance to a number of applications, as it underlies widely used methods like principal components analysis. However, the theoretical results which underly the method fail to apply in the "large n, large p" setting just described. Hence, a thorough study of sample covariance matrices in this setting is needed. Eigenvalues of such large dimensional matrices are of particular interest. The largest and smallest eigenvalues of these matrices are, from the point of view of applications, particularly interesting. The aim of the study is to obtain central limit type theorems for these extreme eigenvalues and use them in Statistics for, for instance, hypothesis testing, having a notion of power, etc... A more applied part of this work concerns efficiently using results from random matrix theory - new and old - to better estimate the eigenvalues of the population covariance with the ultimate aim of better estimating the whole covariance matrix when p and n are both large.Technological progress allows us to store and use massive amounts of data about many aspects of our daily lives. An interesting problem is to use this data to understand how certain traits depend on each other. In the stock market, we might be interested in how the behavior of one stock affects the behavior of another stock;understanding all these interrelationships leads to having a measure of the risk taken by investing in portfolios that use the corresponding stocks. Statisticians have a number of tools to deal with all these interrelationships. We can discover ways to look at the data so that, even if all interrelationships are small or weak, so each trait "should" not help us learn too much about any other trait, we might find combinations of the traits that carry enormous amounts of information. We also know what are typical values for these combinations, so we might be able to detect unusual things in the data by looking at it the right way. Those statistical techniques have very wide applications in various fields of science, ranging from climatology to genetics, image recognition etc... Thousands of research papers are published each year that use these techniques. However, the theory that underlies these statistical techniques was created in an era where massive datasets just did not exist, as they were not storable. This research project is focusing on theories and their applications that are better suited to handle our current massive datasets. The applications should allow us to see structure where the classical tools fail to see any and tell us when there is no structure when the classical tools tell us there is. We also have increasing evidence that our standard tools give us often very inaccurate results about our standard measures of risk or amount of information carried in combination of traits. It seems that risks might be underestimated and amount of information might be overestimated. Part of this research program will be dedicated to measuring how inaccurate the classical results are for large datasets and how can a more relevant theory be used for correcting these inaccuracies.