Record Linkage Across Heterogeneous Data Sources

在根据人口普查数据创建纵向数据的背景下，记录联系指的是在几次人口普查中找到同一个人。最近出现了100%的全国人口普查集合，这使得能够系统地识别和联系不同人口普查中的相同个人，以便创建一个新的个人生命历程信息数据库。主要的挑战是这些历史数据不存在唯一的标识符，因此必须使用所有数据库共有的属性并比较它们的值，以确定两条记录是否引用相同的实体。其他挑战还包括不同的数据库格式、排版错误、数据缺失和报告错误的数据(包括有意和无意的)。此外，并不是所有人口普查的人都会出现在下一次人口普查中，因为死亡和移民会将人口从人口中移走，而出生和移民则会增加新的人口，这些人在上一次人口普查中没有出现，但他们的特征可能与之前的人口普查中的人口特征相似。最后，在连接两个人口普查集合时，处理数百万条记录的叉积将带来巨大的计算挑战。**我研究的总体目标是从历史纵向数据中创造和提取知识。推动我研究的动机是通过计算科学显著促进对加拿大社会的理解。作为这项提议的一部分，我将重点关注两个目标：短期目标是从历史人口普查中创建大规模的纵向数据，这是实现我的目标的复杂但关键的一步；长期目标是从纵向数据中自动提取有用的知识，以丰富我们对加拿大社会的历史和经济的理解。**最近出现的100%数字化的加拿大人口普查收集首次使人们能够大规模、以数据为导向，了解关键的社会变化，如移民、社会流动性、劳动力市场调整和代际不平等。从社会科学的角度来看，主要的挑战是，由于使用的主要是人工链接技术，个人生命历程信息的大规模生成，严重限制了可用于他们研究的数据。我的研究的第一个关键影响*将包括从历史人口普查中自动大规模生成纵向数据。为了实现这一点，我将通过以下五个关键成果显著推进自动记录链接的最新水平，这些成果将成为我实现短期目标的里程碑：更好的特征构建；更严格的候选选择界限；更准确的分类模型；通过使用家庭信息增加链接覆盖面；以及评估和验证历史记录链接的标准化基准。我将与历史和社会科学研究人员分享由此产生的纵向数据；他们一直在等待这种性质和规模的纵向数据，以解决有关社会、历史和经济的紧迫研究问题。第二个关键影响将包括对生成的纵向数据采用知识提取技术，以确定关于加拿大社会的有趣模式。我预计，这一结果还需要设计新的知识提取技术，这些技术更善于识别大规模纵向数据中的模式，这也应该使它们适用于社交网络等相关研究领域。此外，通过与社会研究人员分享这些模式，我们将验证并进一步促进对加拿大社会经济变化的理解。

In the context of creating longitudinal data from census data, record linkage refers to finding the same person across several censuses. The recent emergence of 100 percent national census collections enables a systematic identification and linking of the same individuals across censuses in order to create a new database of individual life-course information. The main challenge is that unique identifiers do not exist for this historical data, thus one must use attributes common to all of the databases and compare their values to determine whether two records refer to the same entity. Other challenges are presented by different database formats, typographical errors, missing data and ill-reported data (both intentional and inadvertent). Furthermore, not everyone in a census is present in the next one because death and emigration remove people from the population, while births and immigration add new people who were not present in the previous census but who may have characteristics similar to those who were present. Finally, processing the cross product of millions of records when linking two census collections presents significant computation challenges. **The overall objective of my research is to create and extract knowledge from historical longitudinal data. The motivation driving my research is to significantly advance the understanding of the Canadian society through computational science. As part of this proposal, I will focus on two goals: the short term goal is the creation of large scale longitudinal data from historical censuses, a complex yet critical step towards my objective; and the long term goal of automatically extracting useful knowledge from the longitudinal data that would enrich our understanding about the history and economics of the Canadian society.**The recent emergence of 100 percent digitized Canadian census collections enables for the first time a large scale, data-driven, understanding of key society changes such as migration, social mobility, labour market adjustments and intergenerational inequality. The main challenge from a social science perspective is the large scale generation of individual life-course information due to the mostly manual linking techniques employed, strongly limiting the data available for their studies. A first key impact*of my research will consist of automatic large scale generation of longitudinal data from historical censuses. To achieve this, I will significantly advance the state-of-the art in the automated record linkage through the five key results that will act as milestone towards my short term goal: better feature construction; tighter bounds for*candidate selection; more accurate classification models; increased linking coverage through the use of family information; and, a standardized benchmark for evaluating and validating historical record linkage. I will share the resulting longitudinal data with researchers in history and the social sciences; they have been waiting for longitudinal data of this nature and scale in order to resolve pressing research questions about society, history and economy. A second key impact will consist of employing knowledge extraction techniques on the generated longitudinal data to identify interesting patterns about the Canadian society. I expect that this result will also require devising new knowledge extraction techniques that are better at identifying patterns in large-scale longitudinal data, which should also make them suitable for related research areas such as social networks. Moreover, by sharing the patterns with social researchers, we will both validate and further advance the understanding of the socio-economic*changes of the Canadian society.