Multiscale Generalized Correlation: A Unified Distance-Based Correlation Measure for Dependency Discovery

长期以来，检测两个数据集之间的关系一直是统计学中最重要的问题之一，也是大数据时代科学发现的基础。通过开发一种开源、健壮、高效和可扩展的统计方法来测试对现代数据的依赖关系，该项目旨在促进对测试依赖关系的理解和实用，解决一些相关的统计推断问题，并加速广泛的数据密集型研究。该项目结合了数学、统计学和计算机科学的基础研究，以进一步开发一个多尺度的广义相关框架，以通过分析大型和复杂的数据来实现发现和决策。正在开发的工具将允许科学家在无数应用中更好地探索和理解高维、非线性和多模式数据。该项目旨在提供一个统一的框架，以便以有效和理论上合理的方式发现观测之间的关系。多尺度广义相关(MGC)是将广义相关的概念与局部性原理相结合的一种优越相关度量，它等于所有可能的局部尺度之间的最优局部相关。通过构建距离相关性和利用最近邻，得到的MGC测试统计量是一种唯一的相关性度量，对于有限二阶矩的所有相关性测试是一致的，并且在各种非线性和高维相关性下表现出比现有方法更好的性能。通过研究基于距离的相关性的理论方面，本项目旨在进一步提高MGC风格的有限样本测试的性能，将其扩展到测试对网络和核数据的相关性，并将其扩展到一般推理问题，如两样本测试、离群点检测和特征筛选等相关性测试，以及在脑活动、网络和文本分析中的应用。总体而言，该项目旨在通过理论进步、全面模拟和真实数据实验，为高维、噪声、大数据中的统计测试建立统一的方法框架。

Detecting relationships between two data sets has long been one of the most important questions in statistics and is fundamental to scientific discovery in the big-data era. By developing an open-source, robust, efficient, and scalable statistical methodology for testing dependence on modern data, this project aims to advance the understanding and utility of testing dependence, tackle a number of related statistical inference questions, and accelerate a broad range of data-intensive research. The project incorporates fundamental research in mathematics, statistics, and computer science to further develop a multiscale generalized correlation framework to enable discovery and decision-making via analysis of large and complex data. The tools under development will allow scientists to better explore and understand high-dimensional, nonlinear, and multi-modal data in a myriad of applications. The project aims to provide a unified framework for discovery of relationships between observations in an efficient and theoretically-sound manner. Combining the notion of generalized correlation with the locality principle, multiscale generalized correlation (MGC) is a superior correlation measure that equals the optimal local correlation among all possible local scales. By building upon distance correlation and making use of nearest neighbors, the resulting MGC test statistic is a unique dependence measure that is consistent for testing against all dependencies with finite second moment, and it exhibits better performance than existing state-of-art methods under a wide variety of nonlinear and high-dimensional dependencies. By investigating the theoretical aspects of distance-based correlations, this project aims to further improve the finite-sample performance of MGC-style tests, extend its capability to testing dependence on network and kernel data, and broaden its utility to general inferential questions beyond dependence testing such as two-sample testing, outlier detection, and feature screening, as well as applications to brain activity, networks, and text analysis. Overall, this project intends to establish a unified methodology framework for statistical testing in high-dimensional, noisy, big data, through theoretical advancements, comprehensive simulations, and real data experiments.