Statistical inference for distributed datasets

技术允许我们存储和描述大量数据。这些数据通常是关于人类的，保留了个人生活的个人痕迹，使其他人比自己更了解客户，并提供了对社会动态的特殊见解。大数据时代已经到来，如此海量的信息无法存储在个人电脑上，而是需要相互连接的机器集群。这种大规模的设置通常基于分布式文件系统（如Hadoop），因为没有一个驱动器可能存储那么多信息。对于“分布式数据”，单个处理器无法访问整个数据集，而是大量处理器各自只能访问数据的一部分。 统计分析可以从大数据中提取知识，但大多数统计模型都是为较小的数据集设计的，假设所有数据都可以从一台计算机中获得。这一假设不适用于大规模样本，只有少数统计方法是直接适用的。对于绝大多数统计工具，除非有人愿意只分析一个大规模数据集中随机选择的一部分，否则就需要新的创新解决方案。 该研究计划的主要目标是使统计工具适应分布式数据的现实。许多统计过程都是基于随机变量的估计规律，但在分布式环境中，不同计算节点之间的通信是一种稀缺资源。共享所有数据不是一种选择，必须在精确度和通信成本之间达成妥协。建立一个变量的利益的法律的估计是一个根本的挑战，这项研究计划提出了一些策略，这样做的分布式架构的约束内。一旦拟议的估计的属性是很好地建立，他们将被用来推广统计程序，如最大似然估计，拟合优度测试和restaurant程序，包括自举。更进一步，还提出了推断数据的多变量依赖结构的策略，即臭名昭著的copula。 通过开发分布式（大）数据的统计方法，该研究项目将提供所有科学都非常需要的工具，就像工业和商业应用一样，这些应用都需要从数据中提取知识。

Technology allows us to store and describe massive amounts of data. These data are often about humans, keeping a personal trace of individual lives, allowing others to know a customer better than himself, and providing a special insight into social dynamics. The era of big data is here, and such massive amounts of information cannot be stored on a personal computer, but rather require clusters of machines that are interconnected. Such large scale setups are typically based on distributed file system (such as Hadoop) because no single drive could possibly store that much information. With “distributed data”, a single processor is unable to access the whole dataset, but instead, a large number of processors each have access to only a part of the data. Statistical analyses can extract knowledge from big data, but most statistical models were designed for smaller datasets, assuming that the whole data was available from one computer. This assumption does not hold for large-scale samples and only a few statistical methods are straightforward to adapt. For the vast majority of statistical tools, unless one is willing to analyse only a randomly selected fraction of a massive dataset, new innovative solutions are required. The main objective of this research program is to adapt statistical tools to the reality of distributed data. Many statistical procedures are based on the estimated law of a random variable, but in a distributed environment, the communication between the different computing nodes is a scarce resource. Sharing all of the data is not an option and a compromise must be struck between precision and communication costs. Building an estimate for the law of a variable of interest is a fundamental challenge and this research program proposes a number of strategies to do so within the constraints of a distributed architecture. Once the properties of the proposed estimates are well established, they will be used to generalize statistical procedures as diverse as maximum likelihood estimation, goodness-of-fit tests and resampling procedures including the bootstrap. Moving things one step further, strategies are also proposed to infer the multivariate dependence structure of the data, the infamous copula. By developing statistical methodology for distributed (big) data, this research project will provide tools that are highly needed in all sciences just as much as in industrial and business applications who all share the need to extract knowledge from data.