Statistical inference for distributed datasets

技术使我们能够存储和描述海量数据。这些数据通常是关于人类的，保留了个人生活的个人痕迹，让其他人比自己更了解客户，并提供了对社会动态的特殊洞察。大数据时代已经到来，如此海量的信息不能存储在个人电脑上，而是需要互联的机器集群。这种大规模的设置通常基于分布式文件系统(如Hadoop)，因为没有单个驱动器可以存储那么多信息。对于分布式数据，单个处理器无法访问整个数据集，相反，大量处理器各自只能访问部分数据。 统计分析可以从大数据中提取知识，但大多数统计模型都是为较小的数据集设计的，假设所有数据都可以从一台计算机上获得。这一假设不适用于大规模样本，只有几种统计方法可以直接适用。对于绝大多数统计工具来说，除非有人愿意只分析海量数据集中随机选择的一小部分，否则就需要新的创新解决方案。 这项研究计划的主要目标是使统计工具适应分布式数据的现实。许多统计过程都是基于随机变量的估计定律，但在分布式环境中，不同计算节点之间的通信是一种稀缺资源。共享所有数据不是一种选择，必须在精度和通信成本之间达成妥协。构建兴趣变量定律的估计是一项基本的挑战，该研究计划提出了许多策略来在分布式体系结构的限制下做到这一点。一旦拟议估计的性质得到很好的确定，它们将被用于推广各种统计程序，如最大似然估计、拟合优度检验和包括Bootstrap在内的重新抽样程序。更进一步，还提出了推断数据的多变量相关性结构的策略，即臭名昭著的Copula。 通过开发分布式(大数据)的统计方法，该研究项目将提供所有科学都非常需要的工具，就像在工业和商业应用中一样，后者都有从数据中提取知识的需求。

Technology allows us to store and describe massive amounts of data. These data are often about humans, keeping a personal trace of individual lives, allowing others to know a customer better than himself, and providing a special insight into social dynamics. The era of big data is here, and such massive amounts of information cannot be stored on a personal computer, but rather require clusters of machines that are interconnected. Such large scale setups are typically based on distributed file system (such as Hadoop) because no single drive could possibly store that much information. With “distributed data”, a single processor is unable to access the whole dataset, but instead, a large number of processors each have access to only a part of the data. Statistical analyses can extract knowledge from big data, but most statistical models were designed for smaller datasets, assuming that the whole data was available from one computer. This assumption does not hold for large-scale samples and only a few statistical methods are straightforward to adapt. For the vast majority of statistical tools, unless one is willing to analyse only a randomly selected fraction of a massive dataset, new innovative solutions are required. The main objective of this research program is to adapt statistical tools to the reality of distributed data. Many statistical procedures are based on the estimated law of a random variable, but in a distributed environment, the communication between the different computing nodes is a scarce resource. Sharing all of the data is not an option and a compromise must be struck between precision and communication costs. Building an estimate for the law of a variable of interest is a fundamental challenge and this research program proposes a number of strategies to do so within the constraints of a distributed architecture. Once the properties of the proposed estimates are well established, they will be used to generalize statistical procedures as diverse as maximum likelihood estimation, goodness-of-fit tests and resampling procedures including the bootstrap. Moving things one step further, strategies are also proposed to infer the multivariate dependence structure of the data, the infamous copula. By developing statistical methodology for distributed (big) data, this research project will provide tools that are highly needed in all sciences just as much as in industrial and business applications who all share the need to extract knowledge from data.