Monitoring and auditing of AI in clinical practice - radiotherapy contour quality assurance

在3D图像上勾画器官轮廓，例如计算机断层扫描（CT）扫描，是规划癌症治疗放射治疗的必要步骤。这是一项耗时且重复的任务。自2017年以来，已经引入了基于AI的方法，在很大程度上实现了这一过程的自动化。然而，临床团队必须检查和编辑这些轮廓或轮廓，以确保它们是准确的。在调试这些工具时，诊所执行准确性的定量评估或尝试测量预期的工作流程效率增益。评估轮廓绘制性能的现有措施和工具依赖于全局、每个器官的定量轮廓绘制相似性措施，将人工智能生成的轮廓与专家手动描绘的轮廓进行比较。一旦将AI解决方案引入轮廓绘制工作流程，通过比较当前临床轮廓绘制实践与前瞻性检查和编辑方法来评估节省的时间。在调试和临床实施后，需要进行持续的质量保证（QA），即“上市后监督”。这包括审核常规临床使用中的用户交互，以确定AI模型在临床实践中的真实性能影响。虽然基于AI的自动轮廓绘制解决方案在调试中进行了定量分析和评估，但没有解决方案可以在临床实践中对其性能进行持续分析。此外，监控检查和编辑的执行情况，并且不被忽视，对于它们的安全部署至关重要。在这个项目中，我们寻求开发一个原型系统，用于自动化，持续的，基于AI的轮廓工具的质量保证和审计。通过使用这样的工具，诊所将能够以更安全的方式实现基于AI的轮廓绘制，并识别此类系统中可能存在的任何限制或偏见。

Outlining organs on a 3D image, such as a computed tomography (CT) scan, is a necessary step in planning radiotherapy for the treatment of cancer. This is a time consuming and repetitive task. Since 2017, AI-based methods have been introduced to automate this process to a large degree. However, the clinical team must check and edit these outlines, or contours, to ensure they are accurate.In commissioning such tools, clinics perform quantitative evaluation of accuracy or attempt to measure the expected workflow efficiency gain. Existing measures and tools for evaluation of contouring performance rely on global, organ per organ, quantitative contouring similarity measures, comparing the AI-generated contours with those manually delineated by an expert. Time saving is assessed by comparing the current clinical contouring practice to the prospective check-and-edit approach once the AI-solution is introduced into the contouring workflow.After commissioning and clinical implementation, an ongoing quality assurance (QA), i.e. 'post-market surveillance', is needed. This includes the auditing of user interactions in routine clinical use to determine the true performance impact of the AI-model in clinical practice. While AI-based auto-contouring solutions are analysed and evaluated quantitatively in commissioning, no solutions exist to perform ongoing analysis of their performance in clinical practice. Furthermore, monitoring that checking-and-editing is being performed, and not being overlooked, is critical to their safe deployment.In this project, we seek to develop a prototype system for automated, and ongoing, quality assurance and audit of AI-based contouring tools. By using such a tool, clinics will be able to implement AI-based contouring in a safer manner and identify any limitations or biases that may exist in such systems.