BIGDATA: Small: DA: Dynamical diffusion map methods for high dimensional data

计划中的研究旨在通过开发高维动态数据的解释、解析和特征提取的新方法，从根本上改变降维技术的现状。这些扩展适合于一个更大的数据分析程序，该程序寻求找到为每种类型的数据集量身定做的内在几何图形。扩散图的发展首先建立了一种从通用数据集恢复几何图形的新方法。最近，我们展示了如何恢复动态系统观测的固有几何，这意味着具有时间顺序的附加结构的数据集。在每一种情况下，侧重于发现特定数据结构的内在几何结构，都以新的降维和降噪算法的形式产生了重大的实际效益。以这一进展为基础，该提议在两个重要方向上极大地扩大了这一努力的影响：(1)克服当前动态扩散图方法中的关键弱点，通过(A)扩展以允许扩散图表示的Laplace-Beltrami算子中的漂移和各向异性，以及(B)通过Hodge星算子将扩散图与离散外部微积分合并，以处理高维动力学；以及(2)发展自动构造的、数据适应的调和(或小波)基，以便捕捉时空数据的基本特征。我们的数据适应构造从先验的空间结构开始，然后将其与数据本身相结合，形成数据适应的空间几何。然后，我们提出了一种新的方法，利用扩散几何，改进了(1)的结果，在适应的几何中找到了代表数据内在特征的对称性。该项目涉及一系列关于计算动力系统理论和方法发展的研究，目的是显著改变分析海量时空数据集的方式。这项研究产生的算法代表了一种全新的时间尺度和空间尺度分离形式，可以将高维动态数据分解为适应动态的部分。这一新方法将处理广泛的时空输入，例如物理实验的高帧速率视频、空间和时间上不规则的地球物理数据库，如海洋、天气或气候时间序列，计量经济学和后勤数据库，以及复杂动态网络上的多变量测量，如生物连接。这些数据来自科学和工程各个领域的问题，特别是物理和生物系统。这些现代高分辨率数据集特别容易受到维度诅咒的影响，由于模型复杂性和数据要求的指数级增长，使得当前的参数统计技术不切实际。我们的方法隐含地消除了冗余，并以自动数据适配的方式选择感兴趣的特征，将统计上有意义的分析的数据需求减少到可行的水平。教育影响包括将研究课题整合到本科和研究生教学中，并通过与物理、生物工程、生物学和医学的合作者联合研究来加强研究基础设施。

The planned research aims to radically transform the current state of the art of dimension reduction, through the development of new approaches to interpretation, resolution, and feature extraction for high-dimensional dynamical data. These extensions fit into a larger program of data analysis which seeks to find the intrinsic geometry tailored to each type of data set. The development of diffusion maps first established a new way to recover geometry from generic data sets. Recently, we showed how to recover the intrinsic geometry for observations of a dynamical system, meaning data sets that have the additional structure of a time ordering. In each case the focus on discovery of the intrinsic geometry for the particular structure of the data resulted in significant practical benefits in the form of new algorithms for dimensionality reduction and noise reduction. With this progress as a foundation, the proposal greatly expands the impact of this effort in two important directions: (1) Overcoming critical weaknesses in the current dynamical diffusion map approach, by (a) extensions to allow drift and anisotropy in the Laplace-Beltrami operator represented by the diffusion map, and (b) merging diffusion maps with discrete exterior calculus through the Hodge star operator to handle higher-dimensional dynamics; and (2) development of an automatically-constructed, data-adapted harmonic (or wavelet) basis in order to capture essential features of spatiotemporal data. Our data-adapted construction starts with an a priori spatial structure and then combines this with the data itself to form the data-adapted spatial geometry. We then propose a novel method of using the diffusion geometry, improved with the results of (1), to find symmetries in the adapted geometry that represent intrinsic features of the data. The project involves a series of investigations in the development of computational dynamical systems theory and methods, with the goal of significantly changing the way massive spatiotemporal data sets are analyzed. The algorithms resulting from the study represent a distinctly new form of time-scale and space-scale separation that can break high-dimensional dynamical data into parts adapted to the dynamics. This new approach will handle a wide range of spatiotemporal inputs, such as high-frame-rate videos of physical experiments, spatially and temporally irregular geophysical databases such as oceanographic, weather or climate time series, econometric and logistical databases, and multivariate measurements on complex dynamic networks such as biological connectomes. The data comes from problems spanning diverse areas of sciences and engineering, with special focus on physical and biological systems. These modern high-resolution data sets are particularly vulnerable to the curse of dimensionality, making current parametric statistical techniques impractical due to exponential increases in model complexity and data requirements. Our approach implicitly eliminates redundancies and selects features of interest in an automatic data-adapted way, reducing the data requirements for statistically significant analyses to feasible levels. Educational impacts include integration of the research topics into undergraduate and graduate teaching, and the enhancement of research infrastructure through joint research with collaborators in physics, bioengineering, biology, and medicine.