Statistical Inferences on Massive Data

该提案计划开发新的统计理论和方法来处理海量数据。提出了理论研究和方法发展的四条相互关联的途径：高维变量选择、大协方差估计、大规模假设检验和非参数统计学习。特别是，提出了新的统计技术来回答以下重要问题：如何利用已有的知识筛选基因和风险因素；折叠凹罚方法的优点是什么；如何估计分类和回归的基准；如何处理孤立点、相关性数据和内生测量；如何利用地理邻域的同质性来增强预测和推断；如何评估风险测量的不确定性；如何进行稀疏主成分分析；如何控制任意依赖下的错误发现率；如何使用非参数方法来增强高维统计学习的灵活性。此外，从金融经济学理论出发，提出了一个新的统计模型，用于估计大协方差矩阵，以便更好地了解风险相关性和更好地评估风险。此外，还介绍了检验内生变量的方法和著名的多因素定价模型，并将对其进行深入研究。大量数据收集已经成为探索科学前沿的例行公事，在一个案例中进行基因组研究，在另一个案例中衡量经济风险。拟议的研究将增进我们对分子机制、生物过程、遗传关联、大脑功能、社会网络、经济和金融风险、供求的了解，从而提高经济和全球竞争力。此外，所提出的新的统计技术还可以应用于其他生物和工程问题。该项目将通过与高年级本科生、研究生和博士后密切合作，将研究和教育结合起来，并通过与行业合作伙伴密切合作，开发公开可用的计算机代码，在坚实的理论支持下处理海量数据，加强学术界和产业界的合作。结果将通过在研讨会、会议、专业协会会议和互联网上的陈述广泛传播。

The proposal plans to develop novel statistical theory and methods for processing massive data. Four interrelated avenues are proposed for theoretical research and methodological developments: High-dimensional variable selection, large covariance estimation, large-scale hypothesis testing, and nonparametric statistical learning. In particular, novel statistical techniques are proposed to answer the following important questions: how to screen genes and risk factors with some acquired knowledge, what are the advantages of folded concave penalized methods, how to estimate the benchmark for classifications and regressions, how to deal with outliers, dependence data, and endogenous measurements, how to use homogeneity of geographical neighborhoods to enhance forecasting and inferences, how to assess uncertainty of risk measurements, how to conduct sparse principal component analysis, how to control the false discovery rates under arbitrary dependence, how to use nonparametric methods to enhance the flexibility of high-dimensional statistical learning. In addition, a novel statistical model, motivated from a financial economics theory, is proposed for estimating large covariance matrices for better understanding risk correlations and for better assessment of risks. The methods for testing the presence of endogenous variables and the celebrated multifactor pricing models are also presented and will be thoroughly investigated. Massive data collections have become routine in exploring the frontiers of science, in one case genomic studies and in another case measuring economic risks. The proposed research will advance our knowledge on understanding molecular mechanisms, biological processes, genetic associations, brain functions, social networks, economic and financial risks, supply and demands, and hence increase economic and global competitiveness. In addition, the proposed novel statistical techniques can be applied to other biological and engineering problems. The project will integrate research and education by working closely with senior undergraduate students, graduate students and postdoctoral fellows, and increase the collaborations between academia and industry by working closely with industrial partners and developing publicly available computer code for processing massive data with sound theoretical supports. The results will be disseminated broadly through presentations at seminars, conferences, professional association meetings, and the internet.