Automatically Leveraging MapReduce Frameworks for Data-Intensive Applications
Automatically Leveraging MapReduce Frameworks for Data-Intensive Applications
复制标题
DOI：
10.1145/3183713.3196891
复制
发表时间：
2018-01
期刊：
影响因子：
--
通讯作者：
Maaz Bin Safeer Ahmad;Alvin Cheung
中科院分区：
文献类型：
--
作者：
Maaz Bin Safeer Ahmad;Alvin Cheung
MapReduce是一种流行的编程范例，用于开发大规模的数据密集型计算。最近开发了许多实现这种范式的框架。然而，要利用这些框架，开发人员必须熟悉它们的api并重写现有代码。我们介绍了Casper，一个自动将顺序Java程序转换为MapReduce范式的新工具。Casper识别潜在的代码片段来重写并翻译它们，分两个步骤：(1)Casper使用程序合成来搜索每个代码片段的程序摘要（即，一个功能规范）。该摘要使用类似于MapReduce范例的高级中间语言表示，并使用定理证明器验证其在语义上与原始摘要等效。(2) Casper从摘要中生成可执行代码，使用Hadoop、Spark或Flink API。我们通过自动将真实的、顺序的Java基准转换为MapReduce来评估Casper。由此产生的基准测试的执行速度比原始测试快48.2倍。
MapReduce is a popular programming paradigm for developing large-scale, data-intensive computation. Many frameworks that implement this paradigm have recently been developed. To leverage these frameworks, however, developers must become familiar with their APIs and rewrite existing code. We present Casper, a new tool that automatically translates sequential Java programs into the MapReduce paradigm. Casper identifies potential code fragments to rewrite and translates them in two steps: (1) Casper uses program synthesis to search for a program summary (i.e., a functional specification) of each code fragment. The summary is expressed using a high-level intermediate language resembling the MapReduce paradigm and verified to be semantically equivalent to the original using a theorem prover. (2) Casper generates executable code from the summary, using either the Hadoop, Spark, or Flink API. We evaluated Casper by automatically converting real-world, sequential Java benchmarks to MapReduce. The resulting benchmarks perform up to 48.2x faster compared to the original.