Making State Explicit for Imperative Big Data Processing
Making State Explicit for Imperative Big Data Processing
复制标题
DOI：
10.17863/cam.41706
复制
发表时间：
2014-06
期刊：
影响因子：
--
通讯作者：
R. Fernandez;Matteo Migliavacca;Evangelia Kalyvianaki;P. Pietzuch
中科院分区：
文献类型：
--
作者：
R. Fernandez;Matteo Migliavacca;Evangelia Kalyvianaki;P. Pietzuch
数据科学家经常使用命令式语言（如Java、Matlab和R）实现机器学习算法。然而，这样的实现未能实现专门的数据并行处理框架的性能和可扩展性。我们的目标是以高吞吐量和低延迟的数据并行方式执行命令式Java程序。这就提出了两个挑战：如何在不影响可伸缩性的情况下支持Java程序的任意可变状态，以及如何在失败后以低开销恢复该状态。我们的想法是从Java程序中推断出状态访问和状态访问的类型，并使用这些信息来生成一个有状态的状态访问图（SDG）。通过显式地将数据与可变状态分离，SDG具有特定的功能来实现这种转换：为了确保可扩展性，如果计算可以完全并行发生，则可以跨节点划分分布式状态;如果这是不可能的，则部分状态为节点提供本地实例进行独立计算，根据应用程序语义进行协调。对于容错，大型内存状态是异步检查点，而无需全局协调。我们表明，几个迫切的在线应用程序的可持续发展目标的性能与现有的数据并行处理框架相匹配。
Data scientists often implement machine learning algorithms in imperative languages such as Java, Matlab and R. Yet such implementations fail to achieve the performance and scalability of specialised data-parallel processing frameworks. Our goal is to execute imperative Java programs in a data-parallel fashion with high throughput and low latency. This raises two challenges: how to support the arbitrary mutable state of Java programs without compromising scalability, and how to recover that state after failure with low overhead. Our idea is to infer the dataflow and the types of state accesses from a Java program and use this information to generate a stateful dataflow graph (SDG). By explicitly separating data from mutable state, SDGs have specific features to enable this translation: to ensure scalability, distributed state can be partitioned across nodes if computation can occur entirely in parallel; if this is not possible, partial state gives nodes local instances for independent computation, which are reconciled according to application semantics. For fault tolerance, large inmemory state is checkpointed asynchronously without global coordination. We show that the performance of SDGs for several imperative online applications matches that of existing data-parallel processing frameworks.