Cuneiform: a Functional Language for Large Scale Scientific Data Analysis
Cuneiform: a Functional Language for Large Scale Scientific Data Analysis
复制标题
DOI：
--
复制
发表时间：
2015
期刊：
影响因子：
--
通讯作者：
Jörgen Brandt;M. Bux;U. Leser
中科院分区：
文献类型：
--
作者：
Jörgen Brandt;M. Bux;U. Leser
一方面分析海量科学数据集的需求，另一方面随着CPU核心数量的增加，分布式计算资源的可用性促进了各种并行、分布式数据分析语言和系统的发展。其中包括数据并行查询语言，如Pig Latin或Spark，以及科学工作流语言，如Swift或Pegasus DAX。数据并行查询语言侧重于数据并行性的开发，而科学工作流语言则侧重于外部工具和库的集成。然而，一种语言，结合了任意工具的轻松集成，被视为黑盒，与充分利用数据并行的能力还不存在。在这里，我们介绍了楔形文字，一种用于大规模科学数据分析的新语言。我们强调其功能方面的一组理想的功能，这样的语言，介绍其语法和语义的例子，并显示其灵活性和简洁性与用例，包括一个复杂的现实生活中的工作流程，从基因组研究领域。楔形文字脚本在基于Hadoop YARN的工作流执行平台Hi-WAY上动态执行。语言Cuneiform，包括编程工具支持，工作流可视化，调试，日志记录和起源跟踪，以及并行执行引擎Hi-WAY完全实现。
The need to analyze massive scientific data sets on the one hand and the availability of distributed compute resources with an increasing number of CPU cores on the other hand have promoted the development of a variety of languages and systems for parallel, distributed data analysis. Among them are data-parallel query languages such as Pig Latin or Spark as well as scientific workflow languages such as Swift or Pegasus DAX. While data-parallel query languages focus on the exploitation of data parallelism, scientific workflow languages focus on the integration of external tools and libraries. However, a language that combines easy integration of arbitrary tools, treated as black boxes, with the ability to fully exploit data parallelism does not exist yet. Here, we present Cuneiform, a novel language for large-scale scientific data analysis. We highlight its functionality with respect to a set of desirable features for such languages, introduce its syntax and semantics by example, and show its flexibility and conciseness with use cases, including a complex real-life workflow from the area of genome research. Cuneiform scripts are executed dynamically on the workflow execution platform Hi-WAY which is based on Hadoop YARN. The language Cuneiform, including tool support for programming, workflow visualization, debugging, logging, and provenance-tracing, and the parallel execution engine Hi-WAY are fully implemented.