CSR: Medium: Collaborative Research: Programming parallel in-memory data-center applications with Piccolo

CSR：媒介：协作研究：使用 Piccolo 对并行内存数据中心应用程序进行编程

• 批准号: 1065169
• 负责人: Jinyang Li
• 金额: $ 52.33万
• 依托单位: New York University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2011
• 资助国家: 美国
• 起止时间: 2011-07-01 至 2015-06-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1065169&HistoricalAwards=false
• 关键词: CSR; Medium; Collaborative; Research; Programming

There is a rising demand to scale application performance by distributingcomputation across many machines in a data-center. It is difficult to writeefficient and robust parallel programs in the data-center setting because programmers need to worry about reducing communication overhead while handling possible machine failures. This project investigates a new data-centric parallel programmingmodel, called Piccolo, that can simplify the construction of in-memorydata-center applications such as PageRank, neural network training etc. In-memory applications can hold all their intermediate states in the aggregatememory of many machines and benefit from sharing these intermediate statesbetween machines during computation. Traditionally, these applicationshave been built using low-level communication-centric primitives such as MPI,resulting in significant programming complexity. The recently popular MapReduce and Dryad also do not fit well with these applicationsbecause their data flow programming model lacks support for shared states.Unlike data flow models, Piccolo explicitly supports the sharing of mutable,distributed states via a key/value table interface. Piccolo makes sharingefficient by optimizing for locality of access to shared tables andautomatically resolving write-write conflicts using user-defined accumulationfunctions. As a result, Piccolo is easy to program for, enables applicationsthat do not fit into MapReduce, and achieves good scalable performance.

通过在数据中心的多台机器上分配计算来扩展应用程序性能的需求不断增加。在数据中心环境中编写高效、健壮的并行程序是很困难的，因为程序员需要在处理可能的机器故障时考虑减少通信开销。 该项目研究了一种新的以数据为中心的并行编程模型Piccolo，它可以简化内存中数据中心应用程序的构建，如PageRank，神经网络训练等。内存中应用程序可以将其所有中间状态保存在多台机器的聚合内存中，并在计算过程中从机器之间共享这些中间状态中受益。 传统上，这些应用程序都是使用低级别的以通信为中心的原语（如MPI）构建的，这导致了显着的编程复杂性。最近流行的MapReduce和Dryad也不适合这些应用程序，因为它们的数据流编程模型缺乏对共享状态的支持。与数据流模型不同，Piccolo通过键/值表接口显式地支持可变的分布式状态的共享。 Piccolo通过优化访问共享表的局部性和使用用户定义的累加函数自动解决写-写冲突来提高共享效率。 因此，Piccolo易于编程，支持不适合MapReduce的应用程序，并实现良好的可扩展性能。