Systematic Evaluation of Assurance Cases

复杂的软件密集型系统导致事故的可能性急剧增加。近年来，我们看到保证案例（ACs）在确保这些系统的安全方面发挥了重要作用。AC提出了一个关于我们想要确保什么的主张，然后提出了明确的推理，有证据支持，我们有足够的信心相信这个主张是正确的。在实践中，这些说法并不精确，而且依赖于什么是“足够好”的社会规范。因此，我们开发评估ACs的合理方法是至关重要的。目前正在进行的研究与ACs的“信心”有关，这主要与结论是否可靠的“衡量”有关。在建议的工作中，AC的“评估”比测量置信度更一般，并且考虑AC质量的多个方面。AC的评估尚未得到应有的重视，并且应该显着提高系统安全性。我研究的5年目标是开发和测试ACs的评估标准和评估过程。它符合我的研究小组的使命，即开发用于认证安全关键型软件密集型系统的有效方法。AC实践的当前状态是使用诸如目标结构符号之类的符号作为AC的基础。我对AC质量评估的研究将考虑当前的AC以及构建AC的新的，更正式的方法，这是我的研究小组在本提案的工作之外开发的。i) ac是巨大而复杂的。ac中的大多数推理是非正式的。iii)我们没有多少指标来评估ac。iv) AC中的基本主张并不精确，并且依赖于社会规范。v)我们必须实现增量保证，即修改现有交流系统以反映各种变化，而不需要重做整个交流系统，也不需要“削弱”交流系统。vi)交流系统可能会受到“确认偏差”的影响。研究成果和方向我们将制定和测试标准，这些标准将构成我们评估方法的基础。在评估AC时，我们需要区分：1)AC的内容；以及2)AC的文档结构/符号。“内容”的初始标准包括：i) AC的令人信服的基础，ii)论证的严谨性，iii)支持完整性论证，iv)可重复性，v)满足证据的接受标准。“结构”的标准包括：1)可追溯性，2)关于变化的健壮性，3)可理解性，4)效率。与AC的作者相比，AC的外部评估者可能使用不同的评估标准。我们将处理这两种情况。对于AC的作者，我们将强调使用成对比较。有效的ac评估是提高和维护复杂软件密集型系统安全性的必要步骤。这项研究将拯救生命。它还将提高加拿大在软件密集型系统认证方面的声誉。

Background The possibility of complex software-intensive systems contributing to accidents has increased dramatically. In recent years we have seen Assurance Cases (ACs) assume a prominent role in assuring safety of these systems. An AC presents a claim about what we want to assure, and then presents explicit reasoning, backed-up by evidence, that we have adequate confidence that the claim is true. In practice, these claims are not precise and rely on societal norms of what is "good enough". It is thus vital that we develop sound ways of evaluating ACs. There is ongoing research related to "confidence" in ACs, which primarily relates to a "measure" of how sound the conclusion is. "Evaluation" of an AC, in the proposed work, is more general than measuring confidence, and considers multiple aspects of the quality of an AC. Evaluation of ACs has not received the attention it warrants, and should significantly improve system safety . Overarching Goal The 5 year goal of my research is to develop and test evaluation criteria and evaluation processes for ACs. It fits within my research group's mission to develop effective methods for certifying safety-critical software-intensive systems. The current state of the practice for ACs is to use a notation such as Goal Structuring Notation as the basis for the AC. My research on evaluation of the quality of ACs will consider current ACs as well as a new, more formal method for building ACs, that my research group is developing outside of the work in this proposal. Challenges i) ACs are enormous and complex. ii) Most reasoning in ACs is informal. iii) We have few metrics to evaluate ACs. iv) The fundamental claim in an AC is not precise, and relies on societal norms. v) We must achieve incremental assurance, i.e., modify existing ACs to reflect all kinds of changes, without redoing the complete AC - and without "weakening" the AC. vi) An AC may be subject to "confirmation bias". Research Outcomes & Direction We will develop and test criteria that will form the basis of our evaluation methods. In evaluating an AC, we need to distinguish between: 1) the content of the AC; and 2) the documentation structure/notation of the AC. Initial criteria for "content" include: i) convincing basis for the AC, ii) rigour of the argument, iii) support for completeness arguments, iv) repeatability, v) satisfaction of acceptance criteria for evidence. Criteria for "structure" include: i) traceability, ii) robustness with respect to change, iii) understandability, iv) efficiency. External evaluators of an AC may use different evaluation criteria compared with the authors of the AC. We will deal with both cases. For authors of the AC, we will emphasize the use of pair-wise comparison.  Impact Effective evaluation of ACs is a necessary step to improve and maintain safety of complex, software-intensive systems. This research will save lives. It will also enhance Canada's reputation in certification of software-intensive systems.