CSR: Medium: Collaborative Research: Programming parallel in-memory data-center applications with Piccolo

点击翻译按钮获取中文摘要

There is a rising demand to scale application performance by distributingcomputation across many machines in a data-center. It is difficult to writeefficient and robust parallel programs in the data-center setting because programmers need to worry about reducing communication overhead while handling possible machine failures. This project investigates a new data-centric parallel programmingmodel, called Piccolo, that can simplify the construction of in-memorydata-center applications such as PageRank, neural network training etc. In-memory applications can hold all their intermediate states in the aggregatememory of many machines and benefit from sharing these intermediate statesbetween machines during computation. Traditionally, these applicationshave been built using low-level communication-centric primitives such as MPI,resulting in significant programming complexity. The recently popular MapReduce and Dryad also do not fit well with these applicationsbecause their data flow programming model lacks support for shared states.Unlike data flow models, Piccolo explicitly supports the sharing of mutable,distributed states via a key/value table interface. Piccolo makes sharingefficient by optimizing for locality of access to shared tables andautomatically resolving write-write conflicts using user-defined accumulationfunctions. As a result, Piccolo is easy to program for, enables applicationsthat do not fit into MapReduce, and achieves good scalable performance.