BIGDATA: Small: DA: Dynamical diffusion map methods for high dimensional data

计划中的研究旨在通过开发新的方法来解释，分辨率和高维动态数据的特征提取，从根本上改变目前的降维技术。这些扩展适合于更大的数据分析程序，该程序旨在找到针对每种数据集的内在几何结构。扩散图的发展首先建立了一种新的方法来恢复几何从一般的数据集。 最近，我们展示了如何恢复动力系统观测的内在几何，这意味着数据集具有时间排序的附加结构。在每种情况下，专注于发现特定数据结构的内在几何结构，以降维和降噪的新算法的形式产生了显着的实际好处。在这一进展的基础上，该提案在两个重要方向上大大扩大了这一努力的影响：（1）克服当前动态扩散图方法中的关键弱点，通过（a）扩展以允许由扩散图表示的拉普拉斯-贝尔特拉米算子中的漂移和各向异性，以及（B）通过Hodge星星算子将扩散映射与离散外部演算合并以处理高维动力学;以及（2）开发自动构造的、数据自适应的谐波（或小波）基，以便捕获时空数据的基本特征。 我们的数据自适应构造从先验空间结构开始，然后将其与数据本身相结合，形成数据自适应空间几何。 然后，我们提出了一种新的方法，使用扩散几何，改进的结果（1），找到适应几何的对称性，代表数据的内在特征。该项目涉及计算动力系统理论和方法发展方面的一系列调查，目标是显著改变分析大量时空数据集的方式。从研究中产生的算法代表了一种独特的新形式的时间尺度和空间尺度分离，可以打破高维动态数据的部分适应的动态。这种新方法将处理各种时空输入，如物理实验的高帧率视频，空间和时间不规则的地球物理数据库，如海洋，天气或气候时间序列，计量经济学和物流数据库，以及对复杂动态网络的多变量测量，如生物连接体。 这些数据来自跨越科学和工程各个领域的问题，特别关注物理和生物系统。 这些现代高分辨率数据集特别容易受到维数灾难的影响，由于模型复杂性和数据要求的指数增加，目前的参数统计技术变得不切实际。 我们的方法隐式地消除冗余，并以自动数据适应的方式选择感兴趣的特征，将统计学显著性分析的数据要求降低到可行的水平。教育影响包括将研究课题整合到本科和研究生教学中，以及通过与物理学，生物工程，生物学和医学合作者的联合研究来增强研究基础设施。

The planned research aims to radically transform the current state of the art of dimension reduction, through the development of new approaches to interpretation, resolution, and feature extraction for high-dimensional dynamical data. These extensions fit into a larger program of data analysis which seeks to find the intrinsic geometry tailored to each type of data set. The development of diffusion maps first established a new way to recover geometry from generic data sets. Recently, we showed how to recover the intrinsic geometry for observations of a dynamical system, meaning data sets that have the additional structure of a time ordering. In each case the focus on discovery of the intrinsic geometry for the particular structure of the data resulted in significant practical benefits in the form of new algorithms for dimensionality reduction and noise reduction. With this progress as a foundation, the proposal greatly expands the impact of this effort in two important directions: (1) Overcoming critical weaknesses in the current dynamical diffusion map approach, by (a) extensions to allow drift and anisotropy in the Laplace-Beltrami operator represented by the diffusion map, and (b) merging diffusion maps with discrete exterior calculus through the Hodge star operator to handle higher-dimensional dynamics; and (2) development of an automatically-constructed, data-adapted harmonic (or wavelet) basis in order to capture essential features of spatiotemporal data. Our data-adapted construction starts with an a priori spatial structure and then combines this with the data itself to form the data-adapted spatial geometry. We then propose a novel method of using the diffusion geometry, improved with the results of (1), to find symmetries in the adapted geometry that represent intrinsic features of the data. The project involves a series of investigations in the development of computational dynamical systems theory and methods, with the goal of significantly changing the way massive spatiotemporal data sets are analyzed. The algorithms resulting from the study represent a distinctly new form of time-scale and space-scale separation that can break high-dimensional dynamical data into parts adapted to the dynamics. This new approach will handle a wide range of spatiotemporal inputs, such as high-frame-rate videos of physical experiments, spatially and temporally irregular geophysical databases such as oceanographic, weather or climate time series, econometric and logistical databases, and multivariate measurements on complex dynamic networks such as biological connectomes. The data comes from problems spanning diverse areas of sciences and engineering, with special focus on physical and biological systems. These modern high-resolution data sets are particularly vulnerable to the curse of dimensionality, making current parametric statistical techniques impractical due to exponential increases in model complexity and data requirements. Our approach implicitly eliminates redundancies and selects features of interest in an automatic data-adapted way, reducing the data requirements for statistically significant analyses to feasible levels. Educational impacts include integration of the research topics into undergraduate and graduate teaching, and the enhancement of research infrastructure through joint research with collaborators in physics, bioengineering, biology, and medicine.