CSR--EHS: Robust Testing by Testing Robustness of Embedded Systems

随着现代嵌入式系统获得更多的功能和复杂性，需要一种新的学科来设计、开发和部署它们。近年来，基于模型的设计范式的思想是开发设计模型，并在实现之前对其进行早期分析、测试和验证。基于模拟的测试确保有限数量的用户定义的系统轨迹满足所需的规格。尽管计算成本低廉的仿真在系统设计中无处不在，但它存在不完备性的问题，因为不可能或不切实际地测试所有系统轨迹。另一方面，验证方法通过显示所有系统轨迹满足期望的性质而享有完备性。对于具有无限可能行为的嵌入式混合系统，详尽的验证似乎非常困难，而基于模拟的测试似乎对我们的系统设计没有信心。除了嵌入式系统的测试和验证之间的差距之外，还有一个更根本的、基本上没有解决的挑战。环境中的不确定性、物理设备中的错误使得整个系统的健壮性成为嵌入式系统中最关键但最不被理解的挑战之一。为分析和测试嵌入式混合系统的健壮性和安全性奠定科学基础并开发方法和算法是一个明显的智力机会。该项目汇集了嵌入式控制、混合系统以及软件监控和测试领域的领先专家，以开发测试嵌入式混合系统健壮性的现代框架的基础。这一建议的核心思想是稳健测试的概念，其中标称测试的稳健性可以计算并用于推断，围绕标称测试的一组轨迹将产生相同的定性行为。通过计算测试的健壮性裕度，该项目探索了如何推断每个测试的健壮性，指导后续测试，估计系统的健壮性，以及使用有限数量的测试提供定义良好的覆盖度量。此外，该项目强调交叉、多部门的研究生教育，并在相关的电气工程和计算机科学课程中强调嵌入式混合系统的测试和健壮性。教育议程是让计算机科学的学生接触到健壮性的概念，并控制学生对软件测试算法的了解。

As modern embedded systems gain more functionality and complexity, there is a need for a novel discipline for their design, development and deployment. In recent years, the idea of the model-based design paradigm is to develop design models and subject them to early analysis, testing, and validation prior to their implementation. Simulation-based testing ensures that a finite number of user-defined system trajectories meet the desired specification. Even though computationally inexpensive simulation is ubiquitous in system design, it suffers from incompleteness, as it is impossible or impractical to test all system trajectories. On the other hand, verification methods enjoy completeness by showing that all system trajectories satisfy the desired property. For embedded hybrid systems with an infinite number of possible behaviors, exhaustive verification seems to be very hard, and simulation-based testing seems to provide no confidence in our system design. In addition to the gap between testing and verification for embedded systems, there is even a more fundamental, and largely unaddressed, challenge. Uncertainty in the environment, errors in physical devices make overall system robustness one of most critical yet least understood challenges in embedded systems. There is a clear intellectual opportunity for laying the scientific foundations and developing methods and algorithms for analyzing and testing the robustness and safety of embedded hybrid systems. This project brings together leading experts in embedded control, hybrid systems, and software monitoring and testing to develop the foundations of a modern framework for testing the robustness of embedded hybrid systems. The central idea that this proposal is centered around is the notion of a robust test, where the robustness of nominal test can be computed and used to infer that a tube of trajectories around the nominal test will yield the same qualitative behavior. By computing the robustness margins of tests, this project explores how to infer how robust each test is, guide subsequent tests, estimate the robustness for the system, as well provide well-defined coverage metrics using finite number of tests. In addition, this project emphasizes cross-cutting, multi-departmental education of graduate students and emphasizes testing and robustness for embedded hybrid systems in relevant electrical engineering and computer science courses. The educational agenda is to expose computer science students to notions of robustness, and control students to software testing algorithms.