Large complex data: divide and recombine (D&R) with RHIPE
Large complex data: divide and recombine (D&R) with RHIPE
复制标题
DOI：
10.1002/sta4.7
复制
发表时间：
2012-01-01
期刊：
影响因子：
1.7
通讯作者：
Cleveland, William S.
中科院分区：
文献类型：
--
作者：
Guha, Saptarshi;Hafen, Ryan;Cleveland, William S.
D&R是分析大型复杂数据的一种新的统计方法。数据被分成几个子集。计算上，每个子集是一个小数据集。将分析方法应用于每个子集，并将每种方法的输出重新组合以形成整个数据的结果。计算可以并行运行，它们之间没有通信，使它们并行，这是最简单的并行处理。使用D&R，数据分析师可以将几乎任何统计或可视化方法应用于大型复杂数据。对整个数据直接应用大多数分析方法要么是不可行的，要么是不切实际的。D&R支持深度分析：全面分析，包括详细数据的可视化，从而最大限度地降低丢失重要信息的风险。我们的D&R研究重点之一是使用统计学来开发分析方法的“最佳”划分和重组程序。另一个是D&R计算环境，它有两个广泛使用的组件，R和Hadoop，以及我们对它们的RHIPE合并。Hadoop是一个分布式数据库和并行计算引擎，可以跨集群执行令人尴尬的并行D&R计算。RHIPE允许完全在R中进行分析，使得使用数据进行编程非常有效。版权所有：John Wiley & Sons， Ltd
D&R is a new statistical approach to the analysis of large complex data. The data are divided into subsets. Computationally, each subset is a small dataset. Analytic methods are applied to each of the subsets, and the outputs of each method are recombined to form a result for the entire data. Computations can be run in parallel with no communication among them, making them embarrassingly parallel, the simplest possible parallel processing. Using D&R, a data analyst can apply almost any statistical or visualization method to large complex data. Direct application of most analytic methods to the entire data is either infeasible, or impractical. D&R enables deep analysis: comprehensive analysis, including visualization of the detailed data, that minimizes the risk of losing important information. One of our D&R research thrusts uses statistics to develop "best" division and recombination procedures for analytic methods. Another is a D&R computational environment that has two widely used components, R and Hadoop, and our RHIPE merger of them. Hadoop is a distributed database and parallel compute engine that executes the embarrassingly parallel D&R computations across a cluster. RHIPE allows analysis wholly from within R, making programming with the data very efficient. Copyright (C) 2012 John Wiley & Sons, Ltd.