Guiding humans to create better labeled datasets for machine learning in biomedical research

项目总结/摘要 机器学习（ML）在过去十年中取得了巨大的进步，这得益于计算和 大型标记数据集的可用性。虽然这些进步对临床和生物医学的影响 研究具有潜在的重要性，这些应用程序面临着独特的挑战，由于难以获得 生物医学专家的标签。此外，ML算法通常无法跨机构进行推广， 由于测量偏差（例如MR扫描仪）或固有的人口统计学或生物学差异导致的数据集 这限制了它们在生物医学科学中的影响。该提案将开发新的 生物医学数据科学家可以将其应用程序用于1. 通过识别为训练ML提供最大益处的最佳标记样本来改进数据标记 算法; 2.提高ML模型在各机构间的泛化能力; 3.在可扩展的 云平台。我们将首先探讨如何改进被称为主动学习的方法， 通过让算法选择解决其弱点的样本来构建标记的数据集， 把这些样本交给专家做标签然后，我们将研究如何选择这些样本， 通过学习不需要的强大模式，提高ML算法在多个机构中的性能 具体到任何一个网站。最后，我们将开发一个可扩展的软件框架， 集成到自己的应用程序中，以利用这些方法，并可以在云上运行 平台，以支持大型数据集的可扩展分析。这项工作将通过以下方式进行： 模拟研究使用了超过280，000个数字病理学图像的人类标记的独特存储库， 多个机构，以及对开发的软件框架的用户研究，重点是 围产期病理学和人类胎盘。软件工具将影响各种生物医学 病理学以外的应用，其中数据标记和多机构研究仍然具有挑战性。

PROJECT SUMMARY / ABSTRACT Machine learning (ML) has seen tremendous advances in the past decade, fueled by growth in computing and the availability of large labeled datasets. While the impact of these advances on clinical and biomedical research are potentially significant, these applications face unique challenges due to the difficulty in acquiring labels from biomedical experts. Furthermore, ML algorithms often fail to generalize across institutions or datasets due to measurement biases (e.g. MR scanners) or intrinsic demographic or biological differences between cohorts / datasets which limits their impact in biomedical science. This proposal will develop new methodology and open-source software that biomedical data scientists can use with their applications to 1. Improve data labeling by identifying the best samples for labeling that provide the most benefit for training ML algorithms; 2. Improve generalization of ML models across institutes; and 3. Perform this work on scalable cloud platforms. We will first explore how to improve upon methods known as active learning that interactively construct labeled datasets by having an algorithm select samples that address its weaknesses and present these samples to an expert for labeling. We will then investigate how these samples can be selected to improve the performance of ML algorithms across multiple institutions by learning robust patterns that are not specific to any one site. Finally, we will develop an extendable software framework that developers can integrate into their own applications to take advantage of these methods, and that can operate on cloud platforms to support scalable analysis of large datasets. This work will be developed through a combination of simulation studies using a unique repository of over 280,000 human markups of digital pathology images at multiple institutions, and also user studies of the developed software frameworks focused on applications in perinatal pathology and the human placenta. The software tools will impact a broad variety of biomedical applications beyond pathology where data labeling and multi-institutional studies remain challenging.