Automating the Integration of EPA Databases

由于政府必须管理的地理范围广泛和任务复杂，其数据以许多不同的方式分开，并在不同的时间由不同的机构收集。由此产生的海量数据异构性意味着无法有效地跨数据源定位、共享或比较数据，更不用说实现计算数据互操作性了。到目前为止，所有包装数据集合的方法，甚至是跨可比数据集创建映射的方法，都需要手工操作。尽管有一些很有前途的工作，但这种映射的自动化创建仍然处于起步阶段，因为等价和差异在所有级别上都表现出来，从单个数据值到元数据，再到围绕整个数据集合的解释性文本。需要更一般的方法来有效地解决这个问题。将数据映射问题视为机器翻译（MT）跨语言映射问题的一个变体，该项目将采用自1990年以来在机器翻译社区开发的新统计算法来发现所有级别的可比数据集之间的对应关系。在机器翻译中，这些技术将跨语言的单词和单词序列对齐。本研究将对这些技术进行调整和扩展，使其在对齐过程中不仅考虑数据值（单词的模拟），还考虑数据格式/正字法、元数据信息和相关文本信息（元数据描述、脚注等），并在单个数据单元级、单元集（列）级和多列级三个层次上进行对齐学习。以前在MT中还没有尝试过多级对准。这些强大的学习技术从未应用于元数据模式集成和/或数据库对齐或包装。如果这些自动学习的映射是有效的，那么数据库包装所需的手工工作量应该会大大减少。将使用两组域数据。空气质量数据将由位于萨克拉门托的加州空气资源委员会的EPA工作人员提供，他们定期将来自加州35个区域空气质量管理区的数据整合到一个全加州范围的数据库中，并将其传递给北卡罗莱纳州的联邦EPA。火灾排放数据将由不同的环保局办公室、美国农业部/林务局和内政部提供。

Due to the wide range of geographic scales and complex tasks the Government must administer, its data is split in many different ways and is collected at different times by different agencies. The resulting massive data heterogeneity means one cannot effectively locate, share, or compare data across sources, let alone achieve computational data interoperability. To date, all approaches to wrap data collections, or even to create mappings across comparable datasets, require manual effort. Despite some promising work, the automated creation of such mappings is still in its infancy, since equivalences and differences manifest themselves at all levels, from individual data values through metadata to the explanatory text surrounding the data collection as a whole. More general methods are required to effectively address this problem. Viewing the data mapping problem as a variant of the cross-language mapping problem of Machine Translation (MT), this project will employ the new statistical algorithms developed since 1990 in the MT community to discover correspondences across comparable datasets at all levels. In MT, the techniques align words and word sequences across languages. This research will adapt and extend the techniques to consider not only data values (the analogue of words) but also data format/orthography, metadata information, and associated textual information (metadata descriptions, footnotes, etc.) in the alignment process, and to perform alignment learning at three levels: individual data cell level, set of cells (column) level, and multi-column level. Multi-level alignment has not been attempted in MT before. These powerful learning techniques have never been applied to metadata schema integration and/or database alignment or wrapping. If these automatically learned mappings are effective, the amount of manual labor required in database wrapping should be significantly reduced. Two sets of domain data will be used. Air quality data will be provided by EPA staff at the California Air Resources Board in Sacramento, who periodically integrate data from some 35 regional Air Quality Management Districts throughout California into a single California-wide database, and pass this along to the Federal EPA in North Carolina. Fire emissions data will be provided by a different set of EPA offices, the USDA/Forest Service, and the Department of Interior.