HNDS-I: From Stacks to Stats: Unlocking International Census Data from Print Volumes

世界上几乎每个国家都进行人口普查。这些人口普查产生了关于人们是谁、他们的教育程度以及他们在哪里工作和生活的信息。人口普查数据对于了解人口如何随时间变化非常重要。然而，这些数据大多只能在图书馆书架上的印刷书籍中找到。从打印页面中获取数据进行分析是一项挑战，因此这些数据很难用于研究。该项目将收集人口普查数据；扫描里面的数据；将数据转换为易于分析的数字格式；让全世界的研究人员都能免费使用。使这些数据易于获取将使研究人员、政策制定者和其他人能够回答有关老龄化、移民、生育率和死亡率等重要问题的问题。该项目以创建IPUMS国际历史地理信息系统（IHGIS）的工作为基础。IHGIS软件目前的工作是从PDF文档中处理和记录数据。该项目通过使用光学字符识别将扫描图像转换为数字数据表，扩展了IHGIS工具，使其能够处理印刷体。为此，讨论了与扫描数据表有关的特殊问题。例如，无法从数字本身判断扫描后的3222应该是8222。该项目开发软件来确定正确的价值；例如，使用数字化表格，软件可以确定在给定年龄组内上学的人数、不上学的人数以及该年龄组的总人数之间的一致性，以确定正确的8,322值。多维表会出现更复杂的问题，例如包含许多不同地理区域的不同年龄组的人所获得的教育水平的表。该软件使用关于表内容的结构化信息来确定特定行和列元素之间的一致性，并执行检查以查找扫描错误。将这些劳动密集型问题自动化将为无法获得数字发布数据的国家和时期提供大量数据。由于地理细节、历史深度和全球覆盖，研究人员将能够使用这些数据来研究随时间的变化以及国家内部和国家之间地区之间的差异。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Nearly every country in the world conducts a census. These censuses generate information about who people are, their education, and where they work and live. Census data are important for understanding how human populations change over time. However, most of these data are found only in printed volumes on library shelves. It is challenging to get data from printed pages ready for analysis, so these data are difficult to use for research. This project will collect census volumes; scan the data inside; convert the data into digital formats that are easy to analyze; and make them freely available to researchers worldwide. Making these data easy to access will allow researchers, policy makers, and others to answer questions about important issues like aging, migration, fertility, and mortality. This project builds on work that created the IPUMS International Historical Geographic Information System (IHGIS). IHGIS software currently works to process and document data from PDF documents. This project extends the IHGIS tools to work with print volumes by using optical character recognition to convert scanned images into digital data tables. To do so, special problems related to scanning data tables are addressed. For example, there is no way to tell from the number itself that a scanned 3,222 should be 8,222 instead. This project develops software to determine the right value; for example, using a digitized table, the software might determine consistency, within a given age group, among the number of people attending school, the number of people not attending school, and the total number of people in the age group to determine the correct 8,322 value. More complicated problems arise with multidimensional tables, such as a table that contains the levels of education attained for people in different age groups across many different geographic regions. The software uses structured information about the content of tables to determine consistency across specific row and column elements and performs the checks to find scanning errors. Automating otherwise labor-intensive problems like these will provide a large collection of data for countries and time periods where digitally published data are not available. Thanks to the geographic detail, historical depth, and global coverage, researchers will be able to use the data to study change over time and differences between places within and between countries.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.