BIGDATA: Small: DA: Dynamical diffusion map methods for high dimensional data

计划中的研究旨在通过开发高维动态数据的解释、解析和特征提取的新方法，从根本上改变降维技术的现状。这些扩展适用于更大的数据分析程序，该程序旨在找到适合每种类型数据集的内在几何结构。扩散映射的发展首次建立了一种从通用数据集恢复几何图形的新方法。最近，我们展示了如何恢复动力系统观测的固有几何，这意味着数据集具有时间顺序的附加结构。在每种情况下，专注于发现数据的特定结构的内在几何，以降维和降噪的新算法的形式产生了重大的实际好处。以这一进展为基础，该提案在两个重要方向上极大地扩展了这一努力的影响：(1)克服当前动态扩散图方法的关键弱点，通过(a)扩展允许扩散图所代表的拉普拉斯-贝尔特拉米算子中的漂移和各向异性，以及(b)通过Hodge星算子将扩散图与离散外部演算合并以处理高维动力学；(2)开发一种自动构建的数据自适应谐波基（或小波基），以捕获时空数据的基本特征。我们的数据适应构造从一个先验的空间结构开始，然后将其与数据本身结合起来，形成数据适应的空间几何。然后，我们提出了一种使用扩散几何的新方法，根据(1)的结果进行改进，以在表示数据固有特征的自适应几何中找到对称性。该项目涉及计算动力系统理论和方法发展的一系列研究，目标是显著改变大量时空数据集的分析方式。该研究产生的算法代表了一种明显的时间尺度和空间尺度分离的新形式，可以将高维动态数据分解为适应动态的部分。这种新方法将处理广泛的时空输入，如物理实验的高帧率视频，空间和时间不规则的地球物理数据库，如海洋学，天气或气候时间序列，计量经济学和物流数据库，以及复杂动态网络（如生物连接体）的多变量测量。这些数据来自不同科学和工程领域的问题，特别关注物理和生物系统。这些现代高分辨率数据集特别容易受到维度的诅咒，由于模型复杂性和数据需求的指数增长，使当前的参数统计技术变得不切实际。我们的方法隐含地消除了冗余，并以自动数据适应的方式选择感兴趣的特征，将统计显著分析的数据需求降低到可行的水平。教育方面的影响包括将研究课题整合到本科和研究生的教学中，以及通过与物理学、生物工程、生物学和医学领域的合作者进行联合研究来加强研究基础设施。

The planned research aims to radically transform the current state of the art of dimension reduction, through the development of new approaches to interpretation, resolution, and feature extraction for high-dimensional dynamical data. These extensions fit into a larger program of data analysis which seeks to find the intrinsic geometry tailored to each type of data set. The development of diffusion maps first established a new way to recover geometry from generic data sets. Recently, we showed how to recover the intrinsic geometry for observations of a dynamical system, meaning data sets that have the additional structure of a time ordering. In each case the focus on discovery of the intrinsic geometry for the particular structure of the data resulted in significant practical benefits in the form of new algorithms for dimensionality reduction and noise reduction. With this progress as a foundation, the proposal greatly expands the impact of this effort in two important directions: (1) Overcoming critical weaknesses in the current dynamical diffusion map approach, by (a) extensions to allow drift and anisotropy in the Laplace-Beltrami operator represented by the diffusion map, and (b) merging diffusion maps with discrete exterior calculus through the Hodge star operator to handle higher-dimensional dynamics; and (2) development of an automatically-constructed, data-adapted harmonic (or wavelet) basis in order to capture essential features of spatiotemporal data. Our data-adapted construction starts with an a priori spatial structure and then combines this with the data itself to form the data-adapted spatial geometry. We then propose a novel method of using the diffusion geometry, improved with the results of (1), to find symmetries in the adapted geometry that represent intrinsic features of the data. The project involves a series of investigations in the development of computational dynamical systems theory and methods, with the goal of significantly changing the way massive spatiotemporal data sets are analyzed. The algorithms resulting from the study represent a distinctly new form of time-scale and space-scale separation that can break high-dimensional dynamical data into parts adapted to the dynamics. This new approach will handle a wide range of spatiotemporal inputs, such as high-frame-rate videos of physical experiments, spatially and temporally irregular geophysical databases such as oceanographic, weather or climate time series, econometric and logistical databases, and multivariate measurements on complex dynamic networks such as biological connectomes. The data comes from problems spanning diverse areas of sciences and engineering, with special focus on physical and biological systems. These modern high-resolution data sets are particularly vulnerable to the curse of dimensionality, making current parametric statistical techniques impractical due to exponential increases in model complexity and data requirements. Our approach implicitly eliminates redundancies and selects features of interest in an automatic data-adapted way, reducing the data requirements for statistically significant analyses to feasible levels. Educational impacts include integration of the research topics into undergraduate and graduate teaching, and the enhancement of research infrastructure through joint research with collaborators in physics, bioengineering, biology, and medicine.