CAREER: Scalable Record Linkage through the Microclustering Property

无论是试图准确估计美国死于败血症的患者人数、国会选区的居民人数，还是死于武装冲突的个人人数，跨多个数据库的重复信息都是一个常见问题。在准确回答这些问题之前，必须以系统和准确的方式从数据库中删除重复的信息。在研究文献中，这个过程通常被称为记录链接、重复数据删除或实体解析。该CAREER奖将为记录链接开发通用方法和可扩展算法，以便实时或近实时地解决紧迫的全球问题。要开发的建模和计算工具将大大增加可分析的数据量。该项目将使研究人员能够解决更广泛的科学问题，并在多个领域推进研究，包括精准医疗、官方统计和人权。为了促进这些进步并鼓励进一步的发展，所有算法都将作为开源软件发布。在教育方面，研究人员将扩大青年机器学习（YiML）计划，使每年50名高中生和50名本科生参加训练营和技能培养讲习班。这将增强未来几年准备学习机器学习的学生的储备。在国际层面，研究者将在国际贝叶斯分析学会会议上讲授讲习班，包括为妇女举办的YiML讲习班。该研究项目将开发灵活的、通用的贝叶斯非参数模型，用于记录链接任务，以准确地传播链接误差的数量。该项目还将开发可扩展的记录链接算法。通过利用聚类、贝叶斯非参数和概率降维算法的最新进展，该项目将推进记录链接的最新技术。所开发的模型和算法将尝试解决微聚类问题，这是本研究的核心。调查员将与领域专家合作，使用来自卫生保健、官方统计和人权的数据集对新方法进行测试。由此得出的估计数可为这些领域的决策者提供有用的信息。

Duplicative information across multiple databases is a common problem, whether one is trying to accurately estimate the number of patients who have died from sepsis in the United States, the number of people who live in a congressional district, or the number of individuals who have died in armed conflicts. Before such questions can be answered accurately, duplicated information from databases must be removed in a systematic and accurate way. In the research literature, this process is commonly known as record linkage, de-duplication, or entity resolution. This CAREER award will develop general methods and scalable algorithms for record linkage so that pressing global issues can be addressed in real time or near real time. The modeling and computational tools to be developed will significantly increase the volume of data that can be analyzed. This project will enable researchers to address a broader range of scientific questions and advance research in multiple domains, including precision medicine, official statistics, and human rights. To facilitate these advances and encourage further development, all algorithms will be released as open source software. In terms of education, the investigator will expand the Youth in Machine Learning (YiML) program to enable 50 high school students and 50 undergraduate students per year to participate in the bootcamp and skills-building workshops offered. This will enhance the pipeline of students prepared to study machine learning in future years. At an international level, the investigator will teach workshops at the International Society for Bayesian Analysis Meeting, including a YiML workshop for women.This research project will develop flexible, general Bayesian nonparametric models for record linkage tasks that propagate the amount of linkage error exactly. The project also will develop scalable record linkage algorithms. By drawing on recent advances in clustering, Bayesian nonparametrics, and probablistic dimension-reduction algorithms, this project will advance the state-of-the-art in record linkage. The models and algorithms to be developed will attempt to solve the microclustering problem, which is at the core of this research. In collaboration with domain experts, the investigator will test the new methods using data sets from health care, official statistics, and human rights. The resulting estimates may provide useful information for policy makers in these areas.