Record Linkage Across Heterogeneous Data Sources

在从人口普查数据创建纵向数据的上下文中，记录链接是指在多个人口普查中找到同一个人。最近出现的100%全国人口普查收集使得系统地识别和连接不同人口普查中的同一个人，以便创建一个新的个人生命历程信息数据库。主要的挑战是这个历史数据不存在唯一标识符，因此必须使用所有数据库共有的属性，并比较它们的值，以确定两个记录是否引用同一个实体。其他挑战包括不同的数据库格式、排版错误、数据缺失和数据报告不实（有意或无意）。此外，并非每次人口普查中的每个人都出现在下一次人口普查中，因为死亡和移民使人口减少，而出生和移民增加了以前人口普查中没有出现的新人口，但他们的特征可能与参加人口普查的人相似。最后，当连接两个人口普查收集时，处理数百万条记录的叉积提出了重大的计算挑战。**我研究的总体目标是从历史纵向数据中创造和提取知识。推动我研究的动机是通过计算科学显著推进对加拿大社会的理解。作为这个提议的一部分，我将重点关注两个目标：短期目标是从历史人口普查中创建大规模的纵向数据，这是实现我的目标的一个复杂但关键的步骤；以及从纵向数据中自动提取有用知识的长期目标，这些知识将丰富我们对加拿大社会历史和经济的理解。**最近出现的100%数字化的加拿大人口普查收集首次使大规模的、数据驱动的、对关键社会变化的理解成为可能，如移民、社会流动、劳动力市场调整和代际不平等。从社会科学的角度来看，主要的挑战是由于所采用的主要是手工链接技术，因此大规模生成个人生命过程信息，这极大地限制了可用于研究的数据。我的研究的第一个关键影响*将包括从历史人口普查中自动大规模生成纵向数据。为了实现这一目标，我将通过五个关键结果显著推进自动化记录链接的最新技术，这些结果将作为实现我的短期目标的里程碑：更好的特征构建；*候选人选择更严格的界限；更准确的分类模型；通过使用家庭信息扩大联系覆盖面；并且，用于评估和验证历史记录链接的标准化基准。我将与历史和社会科学领域的研究人员分享所得的纵向数据；他们一直在等待这种性质和规模的纵向数据，以便解决有关社会、历史和经济的紧迫研究问题。第二个关键影响将包括对生成的纵向数据采用知识提取技术，以确定有关加拿大社会的有趣模式。我希望这一结果还需要设计新的知识提取技术，以便更好地识别大规模纵向数据中的模式，这也应该使它们适合于相关的研究领域，如社交网络。此外，通过与社会研究人员分享这些模式，我们将验证并进一步推进对加拿大社会经济变化的理解。

In the context of creating longitudinal data from census data, record linkage refers to finding the same person across several censuses. The recent emergence of 100 percent national census collections enables a systematic identification and linking of the same individuals across censuses in order to create a new database of individual life-course information. The main challenge is that unique identifiers do not exist for this historical data, thus one must use attributes common to all of the databases and compare their values to determine whether two records refer to the same entity. Other challenges are presented by different database formats, typographical errors, missing data and ill-reported data (both intentional and inadvertent). Furthermore, not everyone in a census is present in the next one because death and emigration remove people from the population, while births and immigration add new people who were not present in the previous census but who may have characteristics similar to those who were present. Finally, processing the cross product of millions of records when linking two census collections presents significant computation challenges. **The overall objective of my research is to create and extract knowledge from historical longitudinal data. The motivation driving my research is to significantly advance the understanding of the Canadian society through computational science. As part of this proposal, I will focus on two goals: the short term goal is the creation of large scale longitudinal data from historical censuses, a complex yet critical step towards my objective; and the long term goal of automatically extracting useful knowledge from the longitudinal data that would enrich our understanding about the history and economics of the Canadian society.**The recent emergence of 100 percent digitized Canadian census collections enables for the first time a large scale, data-driven, understanding of key society changes such as migration, social mobility, labour market adjustments and intergenerational inequality. The main challenge from a social science perspective is the large scale generation of individual life-course information due to the mostly manual linking techniques employed, strongly limiting the data available for their studies. A first key impact*of my research will consist of automatic large scale generation of longitudinal data from historical censuses. To achieve this, I will significantly advance the state-of-the art in the automated record linkage through the five key results that will act as milestone towards my short term goal: better feature construction; tighter bounds for*candidate selection; more accurate classification models; increased linking coverage through the use of family information; and, a standardized benchmark for evaluating and validating historical record linkage. I will share the resulting longitudinal data with researchers in history and the social sciences; they have been waiting for longitudinal data of this nature and scale in order to resolve pressing research questions about society, history and economy. A second key impact will consist of employing knowledge extraction techniques on the generated longitudinal data to identify interesting patterns about the Canadian society. I expect that this result will also require devising new knowledge extraction techniques that are better at identifying patterns in large-scale longitudinal data, which should also make them suitable for related research areas such as social networks. Moreover, by sharing the patterns with social researchers, we will both validate and further advance the understanding of the socio-economic*changes of the Canadian society.