Guiding humans to create better labeled datasets for machine learning in biomedical research

项目摘要/摘要 机器学习(ML)在过去的十年中取得了巨大的进步，这得益于计算机和网络技术的发展 大型标签数据集的可用性。虽然这些进展对临床和生物医学的影响 研究具有潜在的重要意义，这些应用程序面临着独特的挑战，因为获取 来自生物医学专家的标签。此外，ML算法通常不能跨机构或 由于测量偏差(例如磁共振扫描仪)或固有的人口统计学或生物学差异而产生的数据集 在队列/数据集之间，限制了它们在生物医学科学中的影响。这项建议将发展出新的 生物医学数据科学家可以与他们的应用程序一起使用的方法和开源软件到1。 通过确定最好的标注样本来改进数据标注，从而为训练ML提供最大好处 算法；2.改进跨研究所的ML模型的推广；以及3.在Scalable上执行这项工作 云平台。我们将首先探讨如何改进被称为主动学习的方法 通过让算法选择样本来构建标记的数据集，这些样本解决了其弱点并呈现 这些样品交给专家贴标签。然后我们将调查如何选择这些样本来 通过学习不可靠的模式来提高跨多个机构的ML算法的性能 特定于任何一个站点。最后，我们将开发一个可扩展的软件框架，开发人员可以 集成到他们自己的应用程序中，以利用这些方法，并且可以在云上运行 支持大型数据集的可扩展分析的平台。这项工作将通过联合 使用超过28万个数字病理图像的人类标记的独特存储库进行的模拟研究，网址为 多个机构，以及对开发的软件框架的用户研究，重点是 围产期病理学和人类胎盘。这些软件工具将影响广泛的生物医学 在病理学之外的应用，其中数据标记和多机构研究仍然具有挑战性。

PROJECT SUMMARY / ABSTRACT Machine learning (ML) has seen tremendous advances in the past decade, fueled by growth in computing and the availability of large labeled datasets. While the impact of these advances on clinical and biomedical research are potentially significant, these applications face unique challenges due to the difficulty in acquiring labels from biomedical experts. Furthermore, ML algorithms often fail to generalize across institutions or datasets due to measurement biases (e.g. MR scanners) or intrinsic demographic or biological differences between cohorts / datasets which limits their impact in biomedical science. This proposal will develop new methodology and open-source software that biomedical data scientists can use with their applications to 1. Improve data labeling by identifying the best samples for labeling that provide the most benefit for training ML algorithms; 2. Improve generalization of ML models across institutes; and 3. Perform this work on scalable cloud platforms. We will first explore how to improve upon methods known as active learning that interactively construct labeled datasets by having an algorithm select samples that address its weaknesses and present these samples to an expert for labeling. We will then investigate how these samples can be selected to improve the performance of ML algorithms across multiple institutions by learning robust patterns that are not specific to any one site. Finally, we will develop an extendable software framework that developers can integrate into their own applications to take advantage of these methods, and that can operate on cloud platforms to support scalable analysis of large datasets. This work will be developed through a combination of simulation studies using a unique repository of over 280,000 human markups of digital pathology images at multiple institutions, and also user studies of the developed software frameworks focused on applications in perinatal pathology and the human placenta. The software tools will impact a broad variety of biomedical applications beyond pathology where data labeling and multi-institutional studies remain challenging.