NSF FDA/SiR: Development of eeDAP microscopy platform software, validation data, and statistical methods to assess performance of candidate Software as a Medical Device (SaMD)

随着复杂的定量成像显微镜和机器学习（ML）分析软件的出现，计算病理学最近爆发了。FDA的科学家已经指出，需要统计方法和相关数据来设计对组织学幻灯片审查（例如，病理学家，免疫学家，病理学扩展者）和AI/ML模型熟练的读者的研究。因此，这个NSF/FDA驻院学者项目的重点是为读者研究的组成部分开发蓝图，其中读者在有或没有AI/ML模型输出的情况下解释医学图像。该工作包括为FDA设计的光学和数字显微镜设计软件，验证数据和用于评估AI/ML模型的统计方法。NSF/FDA驻校学者项目的第一个目标是验证现有的基于MATLAB的数字和模拟病理学评估环境（eeDAP）平台作为读者研究的工具，并开发免费的开源软件工具来控制eeDAP平台。该团队将在precisionFDA门户网站、多台专家读者的计算机和当地的eeDAP单位之间建立一座“桥梁”。组织学幻灯片注释软件和Python显微镜硬件控制脚本将使用客户端-服务器模型集成，并使用容器化、工作负载编排和覆盖网络来创建可跨多个设备扩展的应用程序。这将在讨论、众包和教学过程中促进与precisionFDA门户网站和专家的互动。该项目的第二个目标是衡量、解释和制定策略以减少读者的可变性。将使用阈值数据的方法。这些方法分别评估专家组每个成员高于和低于阈值的一致性。测试统计量是最终用户和每个专家之间的协议减去每对专家之间的协议，对专家进行平均。该检验统计量的模型将用于生成假设检验。将依次处理多个阈值和相应的假设检验，以调查整个数据范围内的性能。这些工具需要促进可重复的、可推广的、统计上有效的和实用的设备评估，用于评估计划在临床护理中使用的软件。其他诊断成像测试也面临同样的挑战。因此，本建议中开发的方法的适用性远远超出了数字病理学。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Computational pathology has recently exploded with the advent of sophisticated quantitative imaging microscopes and machine learning (ML) analysis software. FDA scientists have indicated that there is a need for statistical methods and relevant data for designing studies of readers skilled in the art of histology slide review (e.g., pathologists, immunologists, pathology extenders) and of AI/ML models. Therefore, this NSF/FDA Scholar-in-Residence project focuses on developing blueprints for components of reader studies in which readers interpret medical images with and without AI/ML model outputs. The work includes design of software for an FDA designed optical and digital microscope, validation data, and statistical methods used for assessment of AI/ML models.The first objective of this NSF/FDA Scholar-in-Residence project is to validate the existing MATLAB based Evaluation Environment for Digital and Analog Pathology (eeDAP) platform as a tool for reader studies and develop free open-source software tools to control the eeDAP platform. The team will build a “bridge” between the precisionFDA portal, multiple expert readers’ computers, and the local eeDAP unit. Histology slide annotation software and Python microscope hardware control scripts will be integrated using a client-server model with the use of containerization, workload orchestration, and overlay networks to create an application that is scalable across multiple devices. This will facilitate interactions with the precisionFDA portal and with experts during discussions, crowdsourcing, and teaching. The second objective of the project is to measure, account for, and develop strategies to reduce reader variability. Methods that threshold the data will be used. The methods separately evaluate agreement above and below the threshold for each member of an expert panel. The test statistic is the agreement between the end user and each expert minus the agreement between each pair of experts, averaged over the experts. Models of this test statistic will be used to generate hypothesis tests. Multiple thresholds and corresponding hypothesis tests will be treated sequentially to investigate performance across the data range. These tools are needed to facilitate reproducible, generalizable, statistically efficient, and practical device assessments in this space for use in evaluation of software planned for use in clinical care. Other diagnostic imaging tests suffer for the same challenges. Therefore, methods developed in this proposal have applicability well beyond digital pathology.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.