Collaborative Research: Statistical Inference for High Dimensional and High Frequency Data

为了实现大数据革命的承诺，当前的项目将重点关注一种常见的数据形式，即高维高频数据（HDHFD），其中数据的快照涉及大量变量，同时每毫秒就有新的数据流。随着数据收集技术的进步，HDHFD出现在从神经科学到患者护理的医疗应用中；财经；地震数据等地球科学；海洋科学，包括渔业和航运；湍流;互联网数据;以及其他数据流可用的领域。首席研究员（pi）的研究重点是如何从复杂的大数据中提取信息，以及如何将数据转化为知识。特别是，该项目旨在开发尖端的数学和统计方法，以揭示控制HDHFD系统的结构。这种结构的特点是跨越时间和维度的依赖网络，分析的作用是提供关于如何在保留数据体系结构的重要特性的同时降低复杂性的指导。本研究的一个组成部分也是关于如何量化HDHFD系统中估计和预测的不确定性。除了开发一般理论之外，该项目还关注财务数据的应用，包括风险管理、预测和投资组合管理。在所有这些金融领域，更精确的估计值（误差范围更小）都将是有用的。主要投资者、监管机构和政策制定者都对其研究结果感兴趣，而且研究结果完全属于公共领域。本项目旨在从多个角度探索高维高频数据（HDHFD）。一个基本的方法是扩展pi的邻近理论。在连续概率下，观测值的结构通常在局部邻域更容易接近（通常是高斯分布），便于统计分析。这是在不改变现有模型的情况下实现的。在对HDHFD数据的因子建模的贡献中，pi将探索时变矩阵分解，包括开发高频数据的奇异值分解（SVD），作为更直接的因子模型路径。我们计划将新的SVD与基于PCA的方法以及L1类型的方法（如非负矩阵分解）进行比较。pi发现了一种观察时间和跨维依赖性的新方法，这种方法最初是由pi与观察到的渐近方差（观察到的AVAR）联系起来开发的。他们现在将研究跨越时间和维度“借用”信息的可能性。该工具将用于矩阵分解，以及为金融过程的漂移部分开发波动性矩阵，这将与他们计划的矩阵分解工作相结合。pi将探索在连续时间内观察到的AVAR的路径，从而提高准确性并简化实现和理论分析。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

To pursue the promise of the big data revolution, the current project will focus on a common form of data, high dimensional high frequency data (HDHFD), where a snapshot of the data involves a large number of variables, and at the same time new data streams in every fraction of milliseconds. With technological advances in data collection, HDHFD occurs in medical applications from neuroscience to patient care; finance and economics; geosciences such as earthquake data; marine science including fishing and shipping; turbulence; internet data; and other areas where data streaming is available. The Principal Investigators' (PIs') research focuses on how to extract information from complex big data and how to turn data into knowledge. In particular, the project seeks to develop cutting-edge mathematics and statistical methodology to uncover the structure governing HDHFD systems. This structure is characterized by a web of dependence across both time and dimension, and the role of analysis is to provide guidance on how to reduce the complexity while retaining the important features of the data architecture. An integral part of this research is also about how to quantify the uncertainty in estimates and forecasts in HDHFD systems. In addition to developing a general theory, the project is concerned with applications to financial data, including risk management, forecasting, and portfolio management. More precise estimators, with improved margins of error, will be useful in all these areas of finance. The results are of interest to main-street investors, regulators and policymakers, and the results are entirely in the public domain. The purpose of this project is to explore high dimensional high frequency data (HDHFD) from several angles. A fundamental approach is to extend the PIs’ contiguity theory. Under a contiguous probability, the structure of the observations is often more accessible (frequently Gaussian) in local neighborhoods, facilitating statistical analysis. This is achieved without altering current models. In a contribution to factor modeling of the HDHFD data, the PIs will explore time-varying matrix decompositions, including the development of a singular value decomposition (SVD) for high frequency data, as a more direct path to a factor model. We plan to compare the new SVD with PCA based methods, as well as L1 type methods such as nonnegative matrix factorization. The PIs have discovered a new way to look at time and cross-dimension dependence, originally developed by the PIs in connection with their observed asymptotic variance (observed AVAR). They will now look into the possibility to "borrow" information across time and dimension. This tool will be used for matrix decompositions, as well as to develop volatility matrices for the drift part of a financial process, which will interface with their planned work on matrix decompositions. The PIs will explore a path to an observed AVAR that takes place in continuous time, thereby improving accuracy and simplifying both implementation and theoretical analysis.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.