Web-Based Infrastructure for Comparison and Validation of Image Computing Methods

描述（由申请人提供）：在过去的几年里，计算算法的验证一直是一个具有挑战性的话题。事实上，医学影像领域的几个国际研讨会开始通过“大挑战”让社区参与进来。一个重大挑战涉及选择驱动生物/科学问题，并要求专家提交他们的最佳结果和方法来解决它。重大挑战通常使用盲验证，以提供无偏见的验证。验证对科学至关重要，因为它在声称其算法的有效性之前向科学家施加了严格的协议。验证还确保算法在临床上可行，并在临床上具有相同的稳健性和准确性。科学界有一个明确的共识，即需要仔细验证。然而，验证仍然是一个挑战，并可能成为一项艰巨的任务，有几个原因。首先，验证实验的总体设计应遵循严格的规则，以符合科学推理。例如，如果配准算法使用界标作为配准的基础，则在验证过程中不应涉及这些相同的界标。第二，测试和训练数据集应该明确识别和分离。测试数据集应仅用于测试目的，而不是调整算法。第三也是最后一点，用来衡量算法误差的指标应该与研究的科学目标相关。例如，如果最大误差是非常高的值，则仅测量分割的所得误差的平均值可能在临床中具有关键影响。 验证仍然是一项艰巨的任务，已经出现了一些工具来帮助科学家进行验证任务。开源Insight Toolkit和Visualization Toolkit提供了现成的医学成像算法，使与其他方法的比较更容易。分割和配准的巨大挑战，如医学图像计算和计算机辅助干预，邀请研究人员相互测试他们的算法，提供一定程度的验证。然而，目前还没有完整的基础设施提供给收集和托管验证工具的研究验证。 该提案的目的是开发一种基础设施，帮助科学家执行验证任务。虽然被认为是全面临床验证的重要因素，但该系统并不旨在进行全面的临床验证，而是帮助研究人员为其临床应用选择最佳工具。该系统名为COVALIC，提供了一个测试和训练数据集的在线存储库，一个用于验证指标的开源框架，以及一个用于托管重大挑战和发布验证结果的基础设施。通过在线系统，研究人员可以方便地使用Web浏览器执行验证任务。此外，COVALIC建立在开放获取和开源的基础上，从而使社区参与进来，并鼓励研究人员分享他们的数据，算法，指标和结果。我们建议开发和测试该系统的帮助下，在该领域的六个专家：临床研究人员，外科医生，计算机科学家和科学研究人员，从而创建一个由最终用户社区设计的系统。 公共卫生相关性：验证是计算方法开发的关键组成部分，通常会带来重大挑战。比较算法性能的主要困难是为训练和测试数据集以及验证指标定义一个共同的参考。另一个挑战是访问其他研究人员的结果和算法。我们建议开发一个直观的基于Web的系统收集，分发和处理验证算法。此外，我们建议开发一个开源框架，用于验证图像处理算法。

DESCRIPTION (provided by applicant): Validation of computing algorithms has been a challenging topic over the last few years. In fact, several international workshops in the medical imaging field started to involve the community through "grand challenges". A grand challenge involves selecting driving biological/scientific problem and asking experts to submit their best results and methods to solve it. Grand challenges often use blind verification in order to provide an unbiased validation. Validation is critical to science because it imposes to scientists a rigorous protocol before claiming the validity of their algorithms. Validation also ensures that algorithms are clinically viable and will perform with the same robustness and accuracy in the clinic. There is a clear consensus among the scientific community that careful validation is needed. However, validation still remains a challenge and can become a laborious task for several reasons. First, the overall design of the validation experiment should follow strict rules in order to be consistent with the scientific reasoning. For instance, if a registration algorithm uses landmarks as a base for registration, these same landmarks should not be involved during the validation process. Second, the testing and training datasets should be clearly identified and separated. The testing datasets should be used only for testing purposes and not to tune the algorithm. Third and last, the metrics used to measure the error of the algorithm should be relevant to the scientific goal of the research. For instance, only measuring the mean value of the resulting error of segmentation could have critical impact in the clinic if the maximum error is a very high value. Validation remains a difficult task and several tools have emerged to help scientists with validation tasks. The open source Insight Toolkit and Visualization Toolkit provide off the shelf algorithms for medical imaging, making comparison with other methods easier. Grand challenges for segmentation and registration, like the ones hosted at the Medical Image Computing and Computer Assisted Intervention, invite researchers to test their algorithms against each other providing a level of validation. However, no complete infrastructure is currently being offer to the research validation for collection and hosting validation tools. The aim of this proposal is to develop an infrastructure to help scientists to perform validation tasks. While considered an important element towards full clinical validation, the system does not aim to perform a full clinical validation, but rather help research choose the best tools for their clinical application. The proposed system, named COVALIC, provides an online repository of testing and training datasets, an open source framework for validation metrics and an infrastructure for hosting grand challenges and publishing validation results. Through the online system, researchers can perform validation tasks from the convenience of a web browser. Furthermore, COVALIC is built upon open access and open source, thus engaging the community in the effort and encouraging researchers to share their data, algorithms, metrics and results. We propose to develop and test the system with the help of six experts in the field: clinical researchers, surgeon, computer scientist, and scientific researchers, thus creating a system designed by the end user community. PUBLIC HEALTH RELEVANCE: Validation is a critical component of the development of computing methods and often present major challenges. The main difficulty in comparing performance of algorithms is to define a common reference for the training and testing datasets as well as validation metrics. The other challenge is to access other researchers' results and algorithms. We propose to develop an intuitive web-based system for collecting, distributing and processing validation algorithms. Additionally, we propose to develop an open-source framework for the validation of image processing algorithms.