Probabilistic Evaluation of Process Model Matching Techniques
Probabilistic Evaluation of Process Model Matching Techniques
复制标题
DOI：
10.1007/978-3-319-46397-1_22
复制
发表时间：
2016-11
期刊：
影响因子：
--
通讯作者：
Elena Kuss;H. Leopold;Han van der Aa;H. Stuckenschmidt;H. Reijers
中科院分区：
文献类型：
--
作者：
Elena Kuss;H. Leopold;Han van der Aa;H. Stuckenschmidt;H. Reijers
流程模型匹配是指自动识别两个流程模型之间对应的活动。它代表了许多高级工艺模型分析技术的基础，例如识别相似工艺部件或工艺模型搜索。一个核心问题是如何评价过程模型匹配技术的性能。通常情况下，即使是人类也不能就一组正确的对应达成一致。然而，目前的评估方法需要一个二进制黄金标准，它清楚地定义了哪些对应是正确的。这种评估方法的缺点是它没有考虑匹配问题的真正复杂性，并且没有公平地评估匹配技术的能力。本文提出了一种新的过程模型匹配技术评价方法。特别是，我们基于对多个注释器的评估来定义精确度和召回率的概率概念。我们使用数据集和2015年过程模型匹配竞赛的结果来评估和比较我们的评估方法。我们发现，我们的概率评估方法为比赛中的匹配技术分配了不同的排名，并允许对它们的表现进行更详细的了解。
Process model matching refers to the automatic identification of corresponding activities between two process models. It represents the basis for many advanced process model analysis techniques such as the identification of similar process parts or process model search. A central problem is how to evaluate the performance of process model matching techniques. Often, not even humans can agree on a set of correct correspondences. Current evaluation methods, however, require a binary gold standard, which clearly defines which correspondences are correct. The disadvantage of this evaluation method is that it does not take the true complexity of the matching problem into account and does not fairly assess the capabilities of a matching technique. In this paper, we propose a novel evaluation method for process model matching techniques. In particular, we build on the assessment of multiple annotators to define probabilistic notions of precision and recall. We use the dataset and the results of the Process Model Matching Contest 2015 to assess and compare our evaluation method. We find that our probabilistic evaluation method assigns different ranks to the matching techniques from the contest and allows to gain more detailed insights into their performance.