Multiscale Generalized Correlation: A Unified Distance-Based Correlation Measure for Dependency Discovery

长期以来，检测两个数据集之间的关系一直是统计学中最重要的问题之一，也是大数据时代科学发现的基础。通过开发一种开源的，强大的，高效的和可扩展的统计方法来测试对现代数据的依赖性，该项目旨在促进对测试依赖性的理解和实用性，解决一些相关的统计推断问题，并加速广泛的数据密集型研究。该项目结合了数学，统计学和计算机科学的基础研究，以进一步开发多尺度广义相关框架，通过分析大型复杂数据进行发现和决策。正在开发的工具将使科学家能够更好地探索和理解无数应用中的高维，非线性和多模态数据。该项目旨在提供一个统一的框架，以有效和理论合理的方式发现观测之间的关系。多尺度广义相关性（MGC）是广义相关性的概念与局部性原理的结合，是一种上级相关性度量，它等于所有可能的局部尺度之间的最优局部相关性。通过建立在距离相关性和利用最近邻，产生的MGC测试统计量是一个独特的依赖性措施，是一致的测试对所有的依赖与有限的二阶矩，它表现出更好的性能比现有的国家的最先进的方法在各种各样的非线性和高维的依赖。通过研究基于距离的相关性的理论方面，该项目旨在进一步提高MGC风格测试的有限样本性能，将其能力扩展到测试对网络和内核数据的依赖性，并将其实用性扩展到依赖性测试之外的一般推理问题，如双样本测试，离群值检测和特征筛选，以及大脑活动，网络和文本分析的应用。总体而言，该项目旨在通过理论进步，综合模拟和真实的数据实验，建立一个统一的方法框架，用于高维，噪声，大数据的统计测试。

Detecting relationships between two data sets has long been one of the most important questions in statistics and is fundamental to scientific discovery in the big-data era. By developing an open-source, robust, efficient, and scalable statistical methodology for testing dependence on modern data, this project aims to advance the understanding and utility of testing dependence, tackle a number of related statistical inference questions, and accelerate a broad range of data-intensive research. The project incorporates fundamental research in mathematics, statistics, and computer science to further develop a multiscale generalized correlation framework to enable discovery and decision-making via analysis of large and complex data. The tools under development will allow scientists to better explore and understand high-dimensional, nonlinear, and multi-modal data in a myriad of applications. The project aims to provide a unified framework for discovery of relationships between observations in an efficient and theoretically-sound manner. Combining the notion of generalized correlation with the locality principle, multiscale generalized correlation (MGC) is a superior correlation measure that equals the optimal local correlation among all possible local scales. By building upon distance correlation and making use of nearest neighbors, the resulting MGC test statistic is a unique dependence measure that is consistent for testing against all dependencies with finite second moment, and it exhibits better performance than existing state-of-art methods under a wide variety of nonlinear and high-dimensional dependencies. By investigating the theoretical aspects of distance-based correlations, this project aims to further improve the finite-sample performance of MGC-style tests, extend its capability to testing dependence on network and kernel data, and broaden its utility to general inferential questions beyond dependence testing such as two-sample testing, outlier detection, and feature screening, as well as applications to brain activity, networks, and text analysis. Overall, this project intends to establish a unified methodology framework for statistical testing in high-dimensional, noisy, big data, through theoretical advancements, comprehensive simulations, and real data experiments.