HNDS-I: From Stacks to Stats: Unlocking International Census Data from Print Volumes

世界上几乎每个国家都进行人口普查。这些人口普查产生了关于人们是谁、他们的教育程度以及他们在哪里工作和生活的信息。人口普查数据对于了解人口如何随时间变化很重要。然而，这些数据中的大多数只在图书馆书架上的印刷卷中找到。从打印的页面中获取数据以供分析是一项具有挑战性的工作，因此这些数据很难用于研究。该项目将收集人口普查卷；扫描其中的数据；将数据转换为易于分析的数字格式；并向世界各地的研究人员免费提供这些数据。使这些数据易于获取，将使研究人员、政策制定者和其他人能够回答有关老龄化、移民、生育和死亡率等重要问题的问题。该项目建立在创建国际历史地理信息系统(IHGIS)的基础上。IHGIS软件目前用于处理和记录PDF文档中的数据。该项目通过使用光学字符识别将扫描的图像转换为数字数据表，将IHGIS工具扩展到处理印刷卷。为此，需要解决与扫描数据表相关的特殊问题。例如，从数字本身无法判断扫描的3,222应该是8,222。该项目开发软件以确定正确的值；例如，使用数字化表格，该软件可以在给定的年龄组内确定上学人数、未上学人数和该年龄组中的总人数之间的一致性，以确定正确的8,322个值。多维表格出现了更复杂的问题，例如，一个表格包含了许多不同地理区域不同年龄段的人所获得的教育水平。该软件使用有关表内容的结构化信息来确定特定行和列元素之间的一致性，并执行检查以查找扫描错误。将这些原本需要劳动密集型的问题自动化，将为无法获得数字发布数据的国家和时间段提供大量数据收集。由于地理细节、历史深度和全球覆盖范围，研究人员将能够使用这些数据来研究随着时间的变化以及国家内部和国家之间的地点差异。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Nearly every country in the world conducts a census. These censuses generate information about who people are, their education, and where they work and live. Census data are important for understanding how human populations change over time. However, most of these data are found only in printed volumes on library shelves. It is challenging to get data from printed pages ready for analysis, so these data are difficult to use for research. This project will collect census volumes; scan the data inside; convert the data into digital formats that are easy to analyze; and make them freely available to researchers worldwide. Making these data easy to access will allow researchers, policy makers, and others to answer questions about important issues like aging, migration, fertility, and mortality. This project builds on work that created the IPUMS International Historical Geographic Information System (IHGIS). IHGIS software currently works to process and document data from PDF documents. This project extends the IHGIS tools to work with print volumes by using optical character recognition to convert scanned images into digital data tables. To do so, special problems related to scanning data tables are addressed. For example, there is no way to tell from the number itself that a scanned 3,222 should be 8,222 instead. This project develops software to determine the right value; for example, using a digitized table, the software might determine consistency, within a given age group, among the number of people attending school, the number of people not attending school, and the total number of people in the age group to determine the correct 8,322 value. More complicated problems arise with multidimensional tables, such as a table that contains the levels of education attained for people in different age groups across many different geographic regions. The software uses structured information about the content of tables to determine consistency across specific row and column elements and performs the checks to find scanning errors. Automating otherwise labor-intensive problems like these will provide a large collection of data for countries and time periods where digitally published data are not available. Thanks to the geographic detail, historical depth, and global coverage, researchers will be able to use the data to study change over time and differences between places within and between countries.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.