RoboTest: Systematic Model-Based Testing and Simulation of Mobile Autonomous Robots

移动和自主机器人在工业和更广泛的社会中发挥着越来越重要的作用；从无人驾驶汽车到家庭辅助，潜在的应用数不胜数。英国政府将机器人技术确定为引领我们未来经济增长的关键技术（tinyurl.com/q8bhcy7）。然而，他们已经认识到，自主机器人是复杂的，通常在不断变化的环境中运行（tinyurl.com/o2u2ts7）。我们如何能确信它们在执行所需的有用功能的同时是安全的？使用测试来检查正确性和安全性是标准做法。机器人的软件开发实践通常包括在机器人制造之前在模拟中进行测试，然后对实际机器人进行测试。模拟有几个好处：我们可以提前测试，测试执行更便宜、更快。例如，仿真不需要机器人进行物理移动。然而，使用真实的机器人进行测试仍然是必要的，因为我们不能确保模拟能够捕获硬件和环境的所有重要方面。在当前的场景中，测试生成通常是手动的；这使得测试昂贵且不可靠，并带来延迟。手动测试生成容易出错，并且可能导致产生错误结论的测试。如果测试错误地指出机器人存在故障，那么开发人员必须花费额外的成本和时间进行调查。如果测试错误地表明机器人的行为符合预期，那么可能会释放一个有缺陷的系统。如果没有系统的方法，测试也可能识别不可行的环境；这样的测试不能用在真正的机器人上。更糟糕的是，手动测试生成限制了生成的测试数量。所有这些都会影响到机器人软件的成本和质量，并且与目前在其他安全关键领域的做法形成鲜明对比，比如交通运输行业，该行业受到严格监管。然而，技术的翻译并非微不足道。例如，缺乏驾驶员纠正错误或对不可预见的情况做出反应，导致自动驾驶汽车的工作条件要大得多。另一个例子是概率算法，它使机器人的行为不确定，因此，很难在测试中重复，更难以描述为正确或不正确。我们将解决所有这些问题与新的自动化测试生成技术的移动和自主机器人。为了使用我们的技术，RoboTest测试人员使用模拟和实现设计中已经使用的熟悉符号构建机器人模型。之后，无需花时间设计模拟场景，只需按下按钮，RoboTest测试器就会生成测试。使用RoboTest，测试更便宜，因为它花费的时间更少，而且更有效，因为RoboTest测试人员可以使用更多的测试，特别是在使用模拟时。为了执行测试，RoboTest测试人员可以从几个采用各种编程方法的模拟器中进行选择。测试的执行也遵循一个按钮。还有一个按钮将模拟转换为部署测试。因此，RoboTest测试人员可以将部署测试的结果追溯到模拟和原始模型。因此，RoboTest测试人员在理解模拟和现实世界之间的现实差距方面处于有利地位。RoboTest测试员知道测试结果是正确的，并且了解测试的结果；例如，它可以保证找到已识别类的错误。因此，RoboTest测试员可以回答这个非常困难的问题：我们测试得够多了吗？总而言之，RoboTest将为移动和自主机器人的测试奠定坚实的基础。RoboTest将使测试在人员努力方面更加高效和有效，因此，实现长期降低成本。

Mobile and autonomous robots have an increasingly important role in industry and the wider society; from driverless vehicles to home assistance, potential applications are numerous. The UK government identified robotics as a key technology that will lead us to future economic growth (tinyurl.com/q8bhcy7). They have recognised, however, that autonomous robots are complex and typically operate in ever-changing environments (tinyurl.com/o2u2ts7). How can we be confident that they perform useful functions, as required, but are safe? It is standard practice to use testing to check correctness and safety. The software-development practice for robotics typically includes testing within simulations, before robots are built, and then testing of the actual robots. Simulations have several benefits: we can test early, and test execution is cheaper and faster. For example, simulation does not require a robot to move physically. Testing with the real robots is, however, still needed, since we cannot be sure that a simulation captures all the important aspects of the hardware and environment.In the current scenario, test generation is typically manual; this makes testing expensive and unreliable, and introduces delays. Manual test generation is error-prone and can lead to tests that produce the wrong verdict. If a test incorrectly states that the robot has a failure, then developers have to investigate, with extra cost and time. If a test incorrectly states that the robot behaves as expected, then a faulty system may be released. Without a systematic approach, tests may also identify infeasible environments; such tests cannot be used with the real robot. To make matters worse, manual test generation limits the number of tests produced. All this affects the cost and quality of robot software, and is in contrast with current practice in other safety-critical areas, like the transport industry, which is highly regulated. Translation of technology, however, is not trivial. For example, lack of a driver to correct mistakes or respond to unforeseen circumstances leads to a much larger set of working conditions for an autonomous vehicle. Another example is provided by probabilistic algorithms, which make the robot behaviour nondeterministic, and so, difficult to repeat in testing and more difficult to characterise as correct or not. We will address all these issues with novel automated test-generation techniques for mobile and autonomous robots. To use our techniques, a RoboTest tester constructs a model of the robot using a familiar notation already employed in the design of simulations and implementations. After that, instead of spending time designing simulation scenarios, the RoboTest tester, with the push of a button, generates tests. With RoboTest, testing is cheaper, since it takes less time, and is more effective, because the RoboTest tester can use many more tests, especially when using a simulation.To execute the tests, the RoboTest tester can choose from a few simulators employing a variety of approaches to programming. Execution of the tests also follows the push of a button. Yet another button translates simulation to deployment tests. So, the RoboTest tester can trace back the results from the deployment tests to the simulation and the original model. So, the RoboTest tester is in a strong position to understand the reality gap between the simulation and the real world.The RoboTest tester knows that the verdicts for the tests are correct, and understands what the testing achieves; for example, it can be guaranteed to find faults of an identified class. So, the RoboTest tester can answer the very difficult question: have we tested enough?In conclusion, RoboTest will move the testing of mobile and autonomous robots onto a sound footing. RoboTest will make testing more efficient and effective in terms of person effort, and so, achieve longer term reduced costs.