Benchflow - A Benchmark for Workflow Management Systems

BenchFlow项目的目标是设计用于评估和比较工作流管理系统（又名业务流程执行引擎）性能的第一个基准。考虑到为建模和执行工作流而提出的大量异构系统和语言，为了确保这个项目的可行性，我们最初将把精力集中在符合WS-BPEL标准的Web服务组合引擎的基准测试上。由于工作流引擎架构固有的复杂性和影响其性能的大量参数，对工作流系统进行基准测试提出了许多科学研究挑战，需要研究一套新的性能评估技术。这些将补充现有的用于基准测试的数据库管理系统（例如，TPC）或编程语言编译器（例如，SPEC），它们已经被很好地理解，并在数据库、编译器和处理器架构行业中推动了非常大的性能改进。工作流系统已经成为构建面向服务的组合应用程序的平台，其性能取决于两个因素：工作流系统本身的性能和组合服务的性能（可能不在工作流的控制范围内）。我们计划使用模型驱动的自/递归测试方法，通过将外部服务实现为流程来消除它们的影响。为基准选择的流程将从真实的WS-BPEL流程中合成。考虑到可以用来观察工作流系统性能的大量指标，我们的目标是提取一组简化的性能指标，以比较不同的引擎以及给定引擎的不同配置。

The goal of the BenchFlow project is to design the first benchmark for assessing and comparing the performance of workflow management systems (aka business process execution engines). Given the large number of heterogeneous systems and languages that have been proposed for modeling and executing workflows, to ensure the feasibility of this project, we will initially focus our efforts on a benchmark for standard WS-BPEL compliant Web service composition engines. Due to the inherent complexity of workflow engine architectures and the very large number of parameters affecting their performance, benchmarking workflow systems poses a number of scientific research challenges that require investigating a novel set of performance evaluation techniques. These will complement existing ones used to benchmark database management systems (e.g., TPC) or programming language compilers (e.g., SPEC), which are already well understood and have contributed to drive a very large performance improvement within the database, compiler and processor architecture industry. Workflow systems have become the platform to build composite service-oriented applications, whose performance depends on two factors: the performance of the workflow system itself and the performance of the composed services (which could lie outside of the control of the workflow). We plan to use a model-driven, self-/recursive testing approach to eliminate the impact of the external services by having them implemented as processes. The processes selected for the benchmark will be synthesized out of real-world WS-BPEL processes. Given the very large number of metrics that can be used to observe the performance of a workflow system, we aim at distilling a reduced set of performance indicators to compare different engines as well as different configurations of a given engine.