Systematic Evaluation of Assurance Cases

背景复杂的软件密集型系统导致事故的可能性急剧增加。近年来，我们看到保障案例(ACS)在确保这些系统的安全方面发挥了突出的作用。AC提出了一个关于我们想要保证的东西的声明，然后提出了有证据支持的明确推理，即我们有足够的信心相信这个声明是真实的。在实践中，这些主张并不准确，而且依赖于什么是“足够好”的社会规范。因此，我们必须找到正确的方法来评估急性冠脉综合征。目前正在进行与ACS的“信心”相关的研究，这主要是关于结论是否合理的“衡量”。在拟议的工作中，对AC的“评估”比测量信心更一般，并考虑AC质量的多个方面。对ACS的评估没有得到应有的重视，应该会显著提高系统安全性。总体目标我研究的五年目标是开发和测试ACS的评估标准和评估过程。它符合我的研究小组的使命，即开发有效的方法来认证安全关键的软件密集型系统。ACS的实践的当前状态是使用诸如目标结构表示法之类的表示法作为AC的基础。我对ACS质量评估的研究将考虑当前的ACS以及一种新的、更正式的构建ACS的方法，我的研究组正在本提案的工作之外开发该方法。挑战I)ACS是巨大而复杂的。Ii)ACS中的大多数推理都是非正式的。Iii)我们几乎没有评估ACS的指标。四)AC中的基本主张并不准确，而且依赖于社会规范。V)我们必须实现增量保证，即修改现有的AC以反映所有类型的变化，而不重做完整的AC--并且不“削弱”AC。Vi)AC可能会受到“确认偏差”的影响。研究结果和方向我们将制定和测试标准，这些标准将成为我们评估方法的基础。在评估AC时，我们需要区分：1)AC的内容；2)AC的文档结构/符号。“内容”的初始标准包括：(一)AC的令人信服的基础，(二)论证的严谨性，(三)对论证的完整性的支持，(四)可重复性，(五)证据接受标准的满意度。“结构”的标准包括：一)可追溯性，二)关于变化的稳健性，三)可理解性，四)效率。与AC的作者相比，AC的外部评估者可以使用不同的评估标准。我们会处理这两宗案件。对于AC的作者，我们将强调使用配对比较。对AC的影响有效性评估是改善和维护复杂的软件密集型系统安全的必要步骤。这项研究将拯救生命。它还将提高加拿大在软件密集型系统认证方面的声誉。

Background The possibility of complex software-intensive systems contributing to accidents has increased dramatically. In recent years we have seen Assurance Cases (ACs) assume a prominent role in assuring safety of these systems. An AC presents a claim about what we want to assure, and then presents explicit reasoning, backed-up by evidence, that we have adequate confidence that the claim is true. In practice, these claims are not precise and rely on societal norms of what is "good enough". It is thus vital that we develop sound ways of evaluating ACs. There is ongoing research related to "confidence" in ACs, which primarily relates to a "measure" of how sound the conclusion is. "Evaluation" of an AC, in the proposed work, is more general than measuring confidence, and considers multiple aspects of the quality of an AC. Evaluation of ACs has not received the attention it warrants, and should significantly improve system safety . Overarching Goal The 5 year goal of my research is to develop and test evaluation criteria and evaluation processes for ACs. It fits within my research group's mission to develop effective methods for certifying safety-critical software-intensive systems. The current state of the practice for ACs is to use a notation such as Goal Structuring Notation as the basis for the AC. My research on evaluation of the quality of ACs will consider current ACs as well as a new, more formal method for building ACs, that my research group is developing outside of the work in this proposal. Challenges i) ACs are enormous and complex. ii) Most reasoning in ACs is informal. iii) We have few metrics to evaluate ACs. iv) The fundamental claim in an AC is not precise, and relies on societal norms. v) We must achieve incremental assurance, i.e., modify existing ACs to reflect all kinds of changes, without redoing the complete AC - and without "weakening" the AC. vi) An AC may be subject to "confirmation bias". Research Outcomes & Direction We will develop and test criteria that will form the basis of our evaluation methods. In evaluating an AC, we need to distinguish between: 1) the content of the AC; and 2) the documentation structure/notation of the AC. Initial criteria for "content" include: i) convincing basis for the AC, ii) rigour of the argument, iii) support for completeness arguments, iv) repeatability, v) satisfaction of acceptance criteria for evidence. Criteria for "structure" include: i) traceability, ii) robustness with respect to change, iii) understandability, iv) efficiency. External evaluators of an AC may use different evaluation criteria compared with the authors of the AC. We will deal with both cases. For authors of the AC, we will emphasize the use of pair-wise comparison.  Impact Effective evaluation of ACs is a necessary step to improve and maintain safety of complex, software-intensive systems. This research will save lives. It will also enhance Canada's reputation in certification of software-intensive systems.