Statistical inference for distributed datasets

点击翻译按钮获取中文摘要

Technology allows us to store and describe massive amounts of data. These data are often about humans, keeping a personal trace of individual lives, allowing others to know a customer better than himself, and providing a special insight into social dynamics. The era of big data is here, and such massive amounts of information cannot be stored on a personal computer, but rather require clusters of machines that are interconnected. Such large scale setups are typically based on distributed file system (such as Hadoop) because no single drive could possibly store that much information. With “distributed data”, a single processor is unable to access the whole dataset, but instead, a large number of processors each have access to only a part of the data. Statistical analyses can extract knowledge from big data, but most statistical models were designed for smaller datasets, assuming that the whole data was available from one computer. This assumption does not hold for large-scale samples and only a few statistical methods are straightforward to adapt. For the vast majority of statistical tools, unless one is willing to analyse only a randomly selected fraction of a massive dataset, new innovative solutions are required. The main objective of this research program is to adapt statistical tools to the reality of distributed data. Many statistical procedures are based on the estimated law of a random variable, but in a distributed environment, the communication between the different computing nodes is a scarce resource. Sharing all of the data is not an option and a compromise must be struck between precision and communication costs. Building an estimate for the law of a variable of interest is a fundamental challenge and this research program proposes a number of strategies to do so within the constraints of a distributed architecture. Once the properties of the proposed estimates are well established, they will be used to generalize statistical procedures as diverse as maximum likelihood estimation, goodness-of-fit tests and resampling procedures including the bootstrap. Moving things one step further, strategies are also proposed to infer the multivariate dependence structure of the data, the infamous copula. By developing statistical methodology for distributed (big) data, this research project will provide tools that are highly needed in all sciences just as much as in industrial and business applications who all share the need to extract knowledge from data.