Collaborative Research: Statistical Inference for High Dimensional and High Frequency Data

为了实现大数据革命的承诺，目前的项目将专注于一种常见的数据形式，即高维高频数据（HDHFD），其中数据的快照涉及大量变量，同时每毫秒都有新的数据流。随着数据收集技术的进步，HDHFD出现在从神经科学到患者护理的医疗应用中;金融和经济学;地球科学，如地震数据;海洋科学，包括渔业和航运;湍流;互联网数据;以及其他数据流可用的领域。首席研究员（PI）的研究重点是如何从复杂的大数据中提取信息，以及如何将数据转化为知识。特别是，该项目旨在开发尖端的数学和统计方法，以揭示管理HDHFD系统的结构。这种结构的特点是跨时间和维度的依赖网络，分析的作用是指导如何在保留数据架构的重要功能的同时降低复杂性。这项研究的一个组成部分也是关于如何量化的不确定性估计和预测HDHFD系统。除了开发一般理论外，该项目还关注金融数据的应用，包括风险管理，预测和投资组合管理。在所有这些金融领域，更精确的估算方法，以及更好的误差幅度，都将是有用的。主流投资者、监管机构和政策制定者对结果感兴趣，结果完全属于公共领域。本项目的目的是从多个角度探索高维高频数据（HDHFD）。一个基本的方法是扩展PI的邻近理论。在连续概率下，观测的结构通常在局部邻域中更容易接近（通常是高斯），便于统计分析。这是在不改变现有模式的情况下实现的。在对HDHFD数据的因子建模的贡献中，PI将探索时变矩阵分解，包括高频数据的奇异值分解（SVD）的开发，作为因子模型的更直接路径。我们计划比较新的SVD与基于PCA的方法，以及L1型方法，如非负矩阵分解。PI发现了一种新的方法来看待时间和跨维依赖性，最初是由PI与他们观察到的渐近方差（观察到的AVAR）相联系。他们现在将研究跨时间和维度“借用”信息的可能性。这一工具将用于矩阵分解，以及为金融过程的漂移部分制定波动矩阵，这将与他们计划的矩阵分解工作相衔接。PI将探索一条通往连续发生的AVAR观测路径，从而提高准确性并简化实施和理论分析。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

To pursue the promise of the big data revolution, the current project will focus on a common form of data, high dimensional high frequency data (HDHFD), where a snapshot of the data involves a large number of variables, and at the same time new data streams in every fraction of milliseconds. With technological advances in data collection, HDHFD occurs in medical applications from neuroscience to patient care; finance and economics; geosciences such as earthquake data; marine science including fishing and shipping; turbulence; internet data; and other areas where data streaming is available. The Principal Investigators' (PIs') research focuses on how to extract information from complex big data and how to turn data into knowledge. In particular, the project seeks to develop cutting-edge mathematics and statistical methodology to uncover the structure governing HDHFD systems. This structure is characterized by a web of dependence across both time and dimension, and the role of analysis is to provide guidance on how to reduce the complexity while retaining the important features of the data architecture. An integral part of this research is also about how to quantify the uncertainty in estimates and forecasts in HDHFD systems. In addition to developing a general theory, the project is concerned with applications to financial data, including risk management, forecasting, and portfolio management. More precise estimators, with improved margins of error, will be useful in all these areas of finance. The results are of interest to main-street investors, regulators and policymakers, and the results are entirely in the public domain. The purpose of this project is to explore high dimensional high frequency data (HDHFD) from several angles. A fundamental approach is to extend the PIs’ contiguity theory. Under a contiguous probability, the structure of the observations is often more accessible (frequently Gaussian) in local neighborhoods, facilitating statistical analysis. This is achieved without altering current models. In a contribution to factor modeling of the HDHFD data, the PIs will explore time-varying matrix decompositions, including the development of a singular value decomposition (SVD) for high frequency data, as a more direct path to a factor model. We plan to compare the new SVD with PCA based methods, as well as L1 type methods such as nonnegative matrix factorization. The PIs have discovered a new way to look at time and cross-dimension dependence, originally developed by the PIs in connection with their observed asymptotic variance (observed AVAR). They will now look into the possibility to "borrow" information across time and dimension. This tool will be used for matrix decompositions, as well as to develop volatility matrices for the drift part of a financial process, which will interface with their planned work on matrix decompositions. The PIs will explore a path to an observed AVAR that takes place in continuous time, thereby improving accuracy and simplifying both implementation and theoretical analysis.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.