haRTStone - Automated Generation of Benchmark Programs for the Evaluation of Analyses and Optimizations for Hard Real-Time Systems

许多嵌入式系统是安全关键的实时系统，必须满足硬截止日期(例如，安全气囊或飞行控制系统)。在设计这样的实时系统时，保证系统的所有任务都能在给定的期限内完成是至关重要的。为此，需要专门的时序分析来检查系统的最坏情况行为，并能够提供这样的保证。在不满足最后期限的情况下，需要应用优化来修改系统的代码，使得时间约束最终得到满足。对硬实时系统的这种分析和优化的研究是一个非常活跃的领域，新的结果定期以非常快的速度呈现出来。当然，对这些分析和优化的评估起着非常重要的作用。如今，评估通常依赖于基准测试，以便将新的分析或优化应用于现有的应用程序、任务或程序代码集合。然而，目前使用的基准非常有限，不足以进行合理和科学的评估，特别是在考虑大规模并行多任务系统的情况下。为了对分析和优化进行有充分依据和可重复的评估，对整个科学界免费提供的普遍适用的基准方法有强烈的需求。基准测试一方面要满足各个研究分支(如可调度性分析、WCET分析、编译器优化)的需要和要求，另一方面又要真实地表示不同的应用领域，如控制或信号处理应用。本项目的目标是实现一个灵活的、可参数的基准测试生成器，以自动、伪随机和可重现的方式生成基准程序。该基准测试生成器将通过生成完整的任务集以及各个任务的实际程序代码来特别涵盖系统和代码级别。为了能够广泛使用生成器，并与任意感兴趣的人和团体进行广泛的合作，该项目将是包容性的，开发的软件将从一开始就公开提供。最后，该项目将产生一种基于基准的评估方法，该方法清楚和可重复地为不同的实时社区描述如何使用基准生成器以获得可信、合理和科学接受的评估结果。

Many embedded systems are safety-critical real-time systems that have to meet hard deadlines (e.g., airbag or flight control systems). When designing such real-time systems, it is of utmost importance to guarantee that all tasks of a system meet their given deadlines. For this purpose, dedicated timing analyses are required that examine the worst-case behavior of a system and are able to provide such guarantees. In the case that deadlines are not met, optimizations need to be applied in order to modify the code of the system such that timing constraints are nevertheless finally met.Research on such analyses and optimizations for hard real-time systems is an extremely lively area where new results are presented regularly and at a very fast pace. Naturally, the evaluation of such analyses and optimizations plays a very important role. Nowadays, evaluation typically relies on benchmarking such that new analyses or optimizations are applied to existing collections of applications, tasks or program codes. The currently used benchmarks are, however, highly limited and not sufficient in order to perform a sound and scientific evaluation, especially if massively parallel multi-task systems are considered.For a well-founded and reproducible evaluation of analyses and optimizations, there is a strong demand for universally applicable benchmark approaches that are freely available for the entire scientific community. Benchmarks should satisfy the needs and requirements of various branches of research (e.g., schedulability analysis, WCET analysis, compiler optimization) on the one hand, but should also, on the other hand, realistically represent different application domains like, e.g., control or signal processing applications.This project aims at the realization of a flexible and parameterizable benchmark generator that produces benchmark programs in an automated, pseudo-randomized and reproducible fashion. This benchmark generator will in particular cover the system and the code level by producing both complete task sets and also actual program codes for the individual tasks. In order to enable a widespread use of the generator and a broad collaboration with arbitrary interested people and groups, this project will be inclusive and the developed software will be openly available right from the beginning. In the end, this project shall lead to a methodology for benchmarking-based evaluation that describes clearly and reproducibly for the different real-time communities, how to use the benchmark generator in order to obtain plausible, sound and scientifically accepted evaluation results.