StaMInA: A Novel Competition to Drive the Comparative Evaluation of State Machine Inference Approaches

软件系统弥漫在现代生活中；它们控制着一切，从电传飞机和金融转账系统，到汽车的ABS制动系统和微波烹饪模式。理解这些复杂系统并确保它们按预期运行的能力至关重要。状态机是一种形式化的图形化表示法，可用于以可访问的方式可视化这些系统的行为。它们还可以用作几种严格的自动化测试和验证技术的基础。目前，状态机必须手动设计和维护。这是一项昂贵且容易出错的任务，特别是在所讨论的系统不断变化的情况下。面对这一挑战，大量的研究致力于用自动化技术来解决这个问题；自动推断软件系统的状态机，通常是从它们的行为样本。这导致了来自世界各地的团体提出的大量解决方案。尽管这些进步是受欢迎的，但它们也引发了一个重要的问题：没有一个公认的过程来评估和比较这些技术。没有证据表明哪种技术比其他技术更好，以及为什么某些技术更胜一筹。这反过来又阻碍了该领域的进一步研究。通过这个项目，我们将通过组织一场国际比赛来解决上述问题，以彻底比较和评估各种状态机推理技术。这项比赛特别新颖，因为它将使用一系列技术来比较不同技术的结果。这将确定(A)哪些技术是最有效的技术，并(B)阐明其有效性的可能原因。预计这项比赛将成为一项常态化的活动，推动该领域的研究。

Software systems pervade modern life; they control everything from fly-by-wire aircraft and financial transfer systems to ABS breaking systems in cars and cooking modes in microwaves. The ability to understand these complex systems, and to make sure that they behave as expected, is crucial. State machines are a formal, diagrammatic notation that can be used to visualise behaviour of these systems in an accessible way. They can also be used as a basis for several rigorous and automated testing and verification techniques.Currently, state machines have to be designed and maintained by hand. This is an expensive and error-prone task, particularly when the system in question is constantly subject to change. Faced with this challenge, a substantial amount of research has been devoted to solving this problem with automated techniques; to automatically infer state machines of software systems, usually from samples of their behaviour. This has resulted in a multitude of proposed solutions from groups around the world.Although these advances are welcome, they have given rise to an important problem: There is no accepted process by which these techniques can be evaluated and compared against each other. There is no evidence to indicate which technique is better than the others, and why certain techniques excel. This in turn hampers further research in the area.With this project, we will address the above problems by organising an international competition to thoroughly compare and evaluate a diverse range of state machine inference techniques. The competition is especially novel because it will employ a range of techniques to compare the results of different techniques against each other. This will identify (a) which techniques are the most effective ones and (b) shed light on the possible reasons for their effectiveness. It is envisaged that this competition will become a regular event, driving research in the area.