Mapreduce: Simplified data processing on large clusters
Mapreduce: Simplified data processing on large clusters
复制标题
DOI：
10.1145/1327452.1327492
复制
发表时间：
2008-01-01
影响因子：
22.7
通讯作者：
Ghemawat, Sanjay
中科院分区：
文献类型：
--
作者：
Dean, Jeffrey;Ghemawat, Sanjay
MapReduce是一种编程模型和相关的实现，用于处理和生成大型数据集，这些数据集适用于各种现实任务。用户根据map和reduce函数指定计算，底层运行时系统自动在大规模机器集群中并行计算，处理机器故障，并调度机器间通信以有效利用网络和磁盘。程序员发现这个系统很容易用途：在过去的四年里，超过一万个不同的MapReduce程序在Google内部实现，平均每天有十万个MapReduce作业在Google集群上执行，每天处理的数据总量超过20 PB。
MapReduce is a programming model and an associated implementation for processing and generating large datasets that is amenable to a broad variety of real-world tasks. Users specify the computation in terms of a map and a reduce function, and the underlying runtime system automatically parallelizes the computation across large-scale clusters of machines, handles machine failures, and schedules inter-machine communication to make efficient use of the network and disks. Programmers find the system easy to use: more than ten thousand distinct MapReduce programs have been implemented internally at Google over the past four years, and an average of one hundred thousand MapReduce jobs are executed on Google's clusters every day, processing a total of more than twenty petabytes of data per day.