Confidence Distribution (CD) and Efficient Approaches for Combining Inferences from Massive Complex Data

现代强大的数据采集技术极大地方便了许多领域的海量数据的收集，这些数据往往具有异构性和复杂的结构。这些数据被用来为科学发现或营销价值做出推断。在实践中，这些数据通常涉及多个研究或分组，每个研究或分组都有自己的数据源，目标是相同的假设或参数。在这种情况下，需要连贯和有效的总体推论方法，以结合个别研究的结果。在处理大数据的不同算法或高性能计算方法的实施和开发中，也出现了对有效组合推理的需求。该项目的目标是将置信度分布的统计学概念应用于开发新的有效方法，以组合来自不同来源的大量复杂数据环境中的多种推断。这种方法应该对许多领域是及时和有用的，包括卫生和医药、市场分析、信息检索、航空安全、国土安全，仅举几例。这个项目建立在所谓的“置信度分布”的最新令人兴奋的发展的基础上，以开发针对海量数据的融合学习。1)有效的非参数融合学习：一种高效的非参数融合学习方法，用于组合来自多个研究的个体推理，具有可实现的算法和充分的理论支持。开发是非参数的和数据驱动的，只需很少的模型假设即可广泛适用。2)用于处理海量和可能的异质数据的拆分-征服-合并方法的有效融合学习：本研究利用并行计算的思想，提出了几种用于分析海量数据的拆分-征服-合并方案。该方法可以大大减少计算开销，但仍然可以得到与整个数据关联的Oracle推理结果。3)预测和测试中的融合学习：本研究发展和推广了基于置信度分布的预测和测试推理的理论框架。这一发展有助于缓解围绕多个测试和模型选择问题的几个众所周知的困难，特别是在大数据设置方面。总体而言，该项目涉及深入的理论发展和复杂数据中的实际问题解决。它非常适合合作研究和学生的积极参与。

Modern powerful data acquisition technology has greatly facilitated the collection of massive data, often with heterogeneous and complex structures, in many domains. Those data are used for drawing inferences for scientific discoveries or marketing values. In practice, the data often involve multiple studies or subpopulations, each with its own data source, targeting the same hypotheses or parameters. In such cases, coherent and efficient overall inference methods for combining findings from individual studies would be needed. The need for efficient combining inferences also arises in the implementation and development of different algorithmic or high-performance computing methods in dealing with big data. The goal of this project is to apply the statistical concept of confidence distribution to developing novel efficient approaches for combining multiple inferences from different sources and in massive complex data settings. Such approaches should be timely and useful for many domains, including health and medicine, market analysis, information retrieval, aviation safety, homeland security, just to name a few. This project builds on the recent exciting developments from the so-called "confidence distribution" to develop fusion learning for massive data. It focuses on three specific developments: 1) Efficient nonparametric fusion learning: an efficient nonparametric approach for combining individual inferences from multiple studies that has implementable algorithms and full theoretical support. The development is nonparametric and data driven, which is broadly applicable with little model assumptions. 2) Efficient fusion learning for the split-conquer-combine approach for handling massive and possibly heterogeneous data: This research utilizes the idea of parallel computing to develop several split-conquer-combine schemes for analysis of massive data. The approach can reduce substantially the computational expenses and yet still achieve the oracle inference outcome associated with the entire data. 3) Fusion learning in prediction and testing: The research develops and generalizes the theoretical framework of inference for prediction and testing based on confidence distributions. This development helps mitigate several well known difficulties surrounding multiple testing, model selection problems, especially in the setting of big data. Overall, the project involves in-depth theoretical development and real problem solving in complex data. It is ideally suited for collaborative research and active participation from students.