TOPIC 454 - SOFTWARE TO EVALUATE ARTIFICIAL INTELLIGENCE/MACHINE LEARNING MEDICAL DEVICES IN ONCOLOGY SETTINGS

在有监督的情况下，基于机器学习的肿瘤人工智能算法会随着时间的推移而退化，并且不能很好地推广。FDA要求对放射学、人工智能产品进行独立测试，以确定其性能。寻找具有足够可变性的代表性数据既耗时又昂贵，而且通常 不能代表临床实际中所体验到的图像质量的多样性。FDA的审查员也不能将新产品与预测设备进行比较。我们提出了一个定义参考数据集和软件的过程，允许基于成像的肿瘤学AI产品的开发人员使用这些数据集进行测试。参考数据集(R.D.)是其肿瘤状况经病理和/或确认的成像数据集 放射检查确认。目标1将确定定义RD的规则和流程。目标2将导致搜索和整理方法学，以改进从我们的研究文库中提取肿瘤学数据。目标3将创建一个端到端的工作流程，用于根据RD策划者测试成像AI模型。最后，我们将在目标4中开发并向FDA提交一份医疗设备开发工具的资格计划。机器学习/人工智能可以改进从医学成像中检测和表征癌症，但没有通用的、基本真实的参考数据来测试和比较新的人工智能算法。

Supervised, Machine Learning based oncologic AI algorithms degrade over time and don’t generalize well. The FDA requires the conduct of standalone testing of radiologic, AI products to characterize their performance. Sourcing representative data with sufficient variability is time consuming, expensive, and often under-represent the variety of image quality experienced in clinical reality. FDA Reviewers also cannot compare new products to predicate devices. We propose a process for defining Reference Datasets and software allowing developers of imaging-based oncology AI products to test using the datasets. A Reference Dataset (R.D.) is an imaging dataset whose oncologic condition is confirmed by pathologic and/or radiographic confirmation. Objective 1 will be defining the rules and process to define R.D.s. Objective 2 will result in search and curation methodology to improve the extraction of oncologic data from our study library. Objective 3 will create an end-end workflow for the testing an imaging AI model against a RD curated. Finally, we will develop and submit a qualification plan for a Medical Device Development Tool to FDA in Objective 4. Machine Learning/AI can improve the detection and characterization of cancers from medical imaging but there are no common, ground-truth reference data to test and compare new AI algorithms.