CSR: Medium: Collaborative Research: Programming parallel in-memory data-center applications with Piccolo

通过在数据中心的多台机器上分布计算来扩展应用程序性能的需求在不断增长。在数据中心环境中编写高效且健壮的并行程序是很困难的，因为程序员需要考虑在处理可能的机器故障的同时减少通信开销。该项目研究了一种新的以数据为中心的并行编程模型，称为Piccolo，它可以简化内存数据中心应用程序的构建，如PageRank、神经网络训练等。内存中应用程序可以将所有中间状态保存在许多机器的聚合内存中，并在计算期间从机器之间共享这些中间状态中获益。传统上，这些应用程序是使用低级的以通信为中心的原语（如MPI）构建的，这导致了极大的编程复杂性。最近流行的MapReduce和Dryad也不适合这些应用程序，因为它们的数据流编程模型缺乏对共享状态的支持。与数据流模型不同，Piccolo显式支持通过键/值表接口共享可变的分布式状态。Piccolo通过优化对共享表访问的局部性，并使用用户定义的累积函数自动解决write-write冲突，使共享变得高效。因此，Piccolo易于编程，支持不适合MapReduce的应用程序，并实现了良好的可扩展性能。

There is a rising demand to scale application performance by distributing computation across many machines in a data-center. It is difficult to write efficient and robust parallel programs in the data-center setting because programmers need to worry about reducing communication overhead while handling possible machine failures. This project investigates a new data-centric parallel programming model, called Piccolo, that can simplify the construction of in-memory data-center applications such as PageRank, neural network training etc. In-memory applications can hold all their intermediate states in the aggregate memory of many machines and benefit from sharing these intermediate states between machines during computation. Traditionally, these applications have been built using low-level communication-centric primitives such as MPI, resulting in significant programming complexity. The recently popular MapReduce and Dryad also do not fit well with these applications because their data flow programming model lacks support for shared states. Unlike data flow models, Piccolo explicitly supports the sharing of mutable, distributed states via a key/value table interface. Piccolo makes sharing efficient by optimizing for locality of access to shared tables and automatically resolving write-write conflicts using user-defined accumulation functions. As a result, Piccolo is easy to program for, enables applications that do not fit into MapReduce, and achieves good scalable performance.