CITCoM: Casual Inference for Testing of Computational Models

计算模型正越来越多地被用于为影响我们所有人的重要问题提供答案。科学家们越来越多地求助于计算模型来模拟各种现象，如药物对生理的影响、社会中疾病的传播或动脉中的血液流动。在公共部门内，计算模型对于能够预测短期天气模式和预测全球变暖的长期影响至关重要。它们对于支持基础设施支出决策也越来越重要;我们在DAFNI项目中的项目合作伙伴正在开发计算建模基础设施，以支持未来十年4600亿英镑的投资。考虑到通常涉及的高风险决策，模型中的错误或“缺陷”可能导致（并已经导致）灾难性后果。这些系统必须经过严格的测试，以尽量减少这种风险。然而，计算模型不适合传统的软件测试和调试技术。它们可以包括大量的参数和配置选项。它们可能需要很长的时间（并且需要大量的计算资源）来执行单个测试运行，这使得运行大量的测试执行是不可行的。它们操作的数据结构可能特别复杂（例如城市或冠状动脉的3D模型），这意味着这些数据结构可能难以合成和检查。最后，如果发现测试运行产生了错误的结果，这些因素可能会使识别模型源代码中的错误变得非常困难。CITCoM基于这样的观察，即挑战在许多方面都源于数据分析。在存在大量输入变量的情况下，存在分析测试行为并确保观察到的行为是由测试的焦点参数引起的（而不是由其他偶然参数偶然引起的）的挑战。匡威的挑战是选择哪些输入需要改变，哪些输入需要控制，以证明给定的输入组合会导致特定的行为，同时保持测试用例的数量最小。如果发生故障，则存在询问数据以定位代码中的故障的挑战。类似的问题出现在广泛的学科中，特别是在流行病学领域--在流行病学领域，人口数据被仔细检查以确定药物治疗或医疗干预的效果。同样，有许多变量在起作用（生活方式，文化背景，遗传特征，习惯）。收集数据可能是昂贵和耗时的。结果可能难以衡量，审查也很复杂。对于这种情况，过去十年中，一种称为因果推断的统计分析方法迅速兴起。这使得统计学家能够以新的强大方式设计和推理流行病学试验和数据，以有效地采样数据，处理缺失的数据属性，并使用现有数据回答“假设”问题，即使相关数据尚未收集。CITCoM将使用这些强大的因果推断分析功能来解决测试计算模型时出现的问题。我们将生成因果推理驱动的自动测试生成技术、测试预言和调试技术。我们将与STFC、DSTL和谢菲尔德大学的DAFNI项目合作伙伴在一系列大型案例研究模型上进行试验和磨练。最终，CITCoM将使我们能够从计算模型中生成、收集和分析证据，以确保它们不包含错误，从而使它们所提供的任何决策都是有根据的和值得信赖的。

Computational models are being used increasingly to offer answers to important questions that affect us all. Scientists are increasingly resorting to computational models to simulate phenomena as diverse as the effects of drugs on a physiology, transmissions of diseases in a society, or the flow of blood through an artery. Within the public sector, computational models are fundamental to enabling the prediction of weather patterns, both in the short term and also to predict the impact of global warming in the longer term. They are also increasingly vital for supporting decisions on infrastructure spend; our project partners in the DAFNI project are developing computational modelling infrastructure to support the investment of £460bn over the course of the coming decade.Given the high-stakes decisions that are usually involved, mistakes or "bugs" in a model can lead (and have led) to disastrous consequences. It is critical that these systems are rigorously tested to minimise this risk.Computational models are however not amenable to traditional software testing and debugging techniques. They can include large numbers of parameters and configuration options. They can take a very long time (and require a lot of computational resources) to execute a single test run, which makes it infeasible to run large numbers of test executions. The data structures that they operate on can be particularly complex (e.g. 3D models of cities or coronary arteries), which means that these can be difficult to synthesise and inspect. Finally, if a test run is found to produce an incorrect result, these factors can make it very difficult to identify where the bug is in the source code of the model.CITCoM is based on the observation that the challenge is in many ways rooted in data-analysis. In the presence of large numbers of input variables, there is the challenge of analysing the tested behaviour and ensuring that the observed behaviour is caused by the parameters that are the focus of the test (and not accidentally caused by other incidental parameters). There is the converse challenge of selecting which inputs need to be varied and which ones need to be controlled to demonstrate that a given combination of inputs causes a particular behaviour whilst keeping the number of test cases this requires to a minimum. If a fault occurs, there is the challenge of interrogating the data to locate the fault in the code.Similar problems arise in a wide range of disciplines, and especially in the field of Epidemiology - where population data are scrutinised to determine the effects of drug treatments or medical interventions. Again, there are many variables at play (lifestyle, cultural background, genetic traits, habits). Collecting data can be expensive and time-consuming. Outcomes can be difficult to measure and complex to scrutinise. For such situations, the last decade has seen the rapid rise of a family of statistical analysis approaches called Causal Inference. This has enabled statisticians to design and reason about epidemiological trials and data in new and powerful ways to efficiently sample data, handle missing data-attributes, and use existing data to answer "what-if" questions, even if the data in question has not been collected yet.CITCoM will use these powerful Causal Inference analysis capabilities to address the problems that arise when testing computational models. We will generate Causal Inference-driven automated test-generation techniques, test oracles, and debugging techniques. These will be trialled and honed on a set of large case-study models in collaboration with our partners on the DAFNI project at STFC, at DSTL, and within The University of Sheffield.Ultimately, CITCoM will enable us to generate, collect, and analyse evidence from computational models to ensure that they do not contain faults, so that any decisions that they feed into are well-founded and trustworthy.