CSR--EHS: Robust Testing by Testing Robustness of Embedded Systems

随着现代嵌入式系统获得更多的功能和复杂性，需要一个新的纪律，他们的设计，开发和部署。近年来，基于模型的设计范式的思想是开发设计模型，并在实现之前对其进行早期分析，测试和验证。 基于仿真的测试确保有限数量的用户定义的系统轨迹满足所需的规格。尽管计算成本低廉的仿真在系统设计中无处不在，但它存在不完整性，因为测试所有系统轨迹是不可能或不切实际的。另一方面，验证方法通过显示所有系统轨迹满足期望的属性来享受完整性。对于具有无限多可能行为的嵌入式混合系统，详尽的验证似乎非常困难，并且基于仿真的测试似乎对我们的系统设计没有信心。 除了嵌入式系统的测试和验证之间的差距之外，还有一个更基本的、很大程度上尚未解决的挑战。环境中的不确定性，物理设备中的错误使整体系统的鲁棒性成为嵌入式系统中最关键但最不了解的挑战之一。有一个明确的知识机会，奠定科学基础，并开发方法和算法，分析和测试嵌入式混合系统的鲁棒性和安全性。该项目汇集了嵌入式控制、混合系统以及软件监控和测试方面的领先专家，为测试嵌入式混合系统的鲁棒性奠定了现代框架的基础。这个建议的中心思想是围绕着一个强大的测试的概念，其中标称测试的鲁棒性可以被计算出来，并用来推断出一个管的轨道周围的标称测试将产生相同的定性行为。通过计算测试的鲁棒性裕度，该项目探讨了如何推断每个测试的鲁棒性，指导后续测试，估计系统的鲁棒性，以及使用有限数量的测试提供定义良好的覆盖度量。 此外，该项目强调研究生的跨学科，多部门教育，并强调相关电气工程和计算机科学课程中嵌入式混合系统的测试和鲁棒性。教育议程是让计算机科学专业的学生接触鲁棒性的概念，并控制学生进行软件测试算法。

As modern embedded systems gain more functionality and complexity, there is a need for a novel discipline for their design, development and deployment. In recent years, the idea of the model-based design paradigm is to develop design models and subject them to early analysis, testing, and validation prior to their implementation. Simulation-based testing ensures that a finite number of user-defined system trajectories meet the desired specification. Even though computationally inexpensive simulation is ubiquitous in system design, it suffers from incompleteness, as it is impossible or impractical to test all system trajectories. On the other hand, verification methods enjoy completeness by showing that all system trajectories satisfy the desired property. For embedded hybrid systems with an infinite number of possible behaviors, exhaustive verification seems to be very hard, and simulation-based testing seems to provide no confidence in our system design. In addition to the gap between testing and verification for embedded systems, there is even a more fundamental, and largely unaddressed, challenge. Uncertainty in the environment, errors in physical devices make overall system robustness one of most critical yet least understood challenges in embedded systems. There is a clear intellectual opportunity for laying the scientific foundations and developing methods and algorithms for analyzing and testing the robustness and safety of embedded hybrid systems. This project brings together leading experts in embedded control, hybrid systems, and software monitoring and testing to develop the foundations of a modern framework for testing the robustness of embedded hybrid systems. The central idea that this proposal is centered around is the notion of a robust test, where the robustness of nominal test can be computed and used to infer that a tube of trajectories around the nominal test will yield the same qualitative behavior. By computing the robustness margins of tests, this project explores how to infer how robust each test is, guide subsequent tests, estimate the robustness for the system, as well provide well-defined coverage metrics using finite number of tests. In addition, this project emphasizes cross-cutting, multi-departmental education of graduate students and emphasizes testing and robustness for embedded hybrid systems in relevant electrical engineering and computer science courses. The educational agenda is to expose computer science students to notions of robustness, and control students to software testing algorithms.