Collaborative Research: Statistical Inference for High Dimensional and High Frequency Data

为了实现大数据革命的承诺，目前的项目将专注于一种常见的数据形式-高维高频数据(HDHFD)，其中数据的快照涉及大量变量，同时每隔几毫秒就有新的数据流。随着数据收集技术的进步，HDHFD出现在从神经科学到病人护理、金融和经济、地震数据等地球科学、包括渔业和航运在内的海洋科学、湍流、互联网数据以及其他可以使用数据流的领域的医疗应用中。首席调查员的研究重点是如何从复杂的大数据中提取信息，如何将数据转化为知识。特别是，该项目寻求开发尖端数学和统计方法，以揭示管理HDHFD系统的结构。这种结构的特点是跨越时间和维度的依赖网络，分析的作用是提供指导，说明如何在保留数据体系结构的重要功能的同时降低复杂性。这项研究的一个组成部分也是关于如何量化HDHFD系统中估计和预测的不确定性。除了开发一般理论之外，该项目还关注财务数据的应用，包括风险管理、预测和投资组合管理。在所有这些金融领域，更精确的估计器和更高的误差幅度将是有用的。研究结果是普通投资者、监管者和政策制定者感兴趣的结果，而且结果完全是公开的。本项目的目的是从多个角度探索高维高频数据。一个基本的方法是扩展PI的邻接性理论。在连续概率下，观测的结构通常在当地社区更容易获得(通常是高斯的)，便于统计分析。这是在不改变现有模式的情况下实现的。作为对HDHFD数据的因素建模的贡献，PIS将探索时变矩阵分解，包括开发高频数据的奇异值分解(SVD)，作为建立因素模型的更直接途径。我们计划将新的奇异值分解方法与基于主成分分析的方法以及L1类方法(如非负矩阵分解)进行比较。PI已经发现了一种新的方法来看待时间和跨维相关性，最初是由PI根据其观察到的渐近方差(观察到的AVAR)开发的。他们现在将研究跨时间和跨维度“借用”信息的可能性。这一工具将用于矩阵分解，以及为财务流程的漂移部分编制波动率矩阵，这将与他们计划的矩阵分解工作相结合。PIS将探索一条连续发生的观察到的AVAR的途径，从而提高准确性并简化实施和理论分析。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

To pursue the promise of the big data revolution, the current project will focus on a common form of data, high dimensional high frequency data (HDHFD), where a snapshot of the data involves a large number of variables, and at the same time new data streams in every fraction of milliseconds. With technological advances in data collection, HDHFD occurs in medical applications from neuroscience to patient care; finance and economics; geosciences such as earthquake data; marine science including fishing and shipping; turbulence; internet data; and other areas where data streaming is available. The Principal Investigators' (PIs') research focuses on how to extract information from complex big data and how to turn data into knowledge. In particular, the project seeks to develop cutting-edge mathematics and statistical methodology to uncover the structure governing HDHFD systems. This structure is characterized by a web of dependence across both time and dimension, and the role of analysis is to provide guidance on how to reduce the complexity while retaining the important features of the data architecture. An integral part of this research is also about how to quantify the uncertainty in estimates and forecasts in HDHFD systems. In addition to developing a general theory, the project is concerned with applications to financial data, including risk management, forecasting, and portfolio management. More precise estimators, with improved margins of error, will be useful in all these areas of finance. The results are of interest to main-street investors, regulators and policymakers, and the results are entirely in the public domain.The purpose of this project is to explore high dimensional high frequency data (HDHFD) from several angles. A fundamental approach is to extend the PIs’ contiguity theory. Under a contiguous probability, the structure of the observations is often more accessible (frequently Gaussian) in local neighborhoods, facilitating statistical analysis. This is achieved without altering current models. In a contribution to factor modeling of the HDHFD data, the PIs will explore time-varying matrix decompositions, including the development of a singular value decomposition (SVD) for high frequency data, as a more direct path to a factor model. We plan to compare the new SVD with PCA based methods, as well as L1 type methods such as nonnegative matrix factorization. The PIs have discovered a new way to look at time and cross-dimension dependence, originally developed by the PIs in connection with their observed asymptotic variance (observed AVAR). They will now look into the possibility to "borrow" information across time and dimension. This tool will be used for matrix decompositions, as well as to develop volatility matrices for the drift part of a financial process, which will interface with their planned work on matrix decompositions. The PIs will explore a path to an observed AVAR that takes place in continuous time, thereby improving accuracy and simplifying both implementation and theoretical analysis.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.