Cleaning and Analysis of Large Uncertain and Inconsistent Data Sources

大量不确定且不一致的数据源的清理和分析

• 批准号: RGPIN-2014-06143
• 负责人: Ilyas, Ihab
• 金额: $ 5.54万
• 依托单位: University of Waterloo
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2018
• 资助国家: 加拿大
• 起止时间: 2018-01-01 至 2019-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=654237
• 关键词: Cleaning; Analysis; Large; Uncertain; Inconsistent

Data generated by modern applications such as object tracking, sensor networks, health record management, and Web data integration involves uncertainty, and various anomalies such as missing values and duplication. While many research efforts have been focusing on data cleaning and dealing with inconsistent databases, very limited research has been adopted in real settings for various technical and practical challenges around increasing and extracting value from large dirty data sets. To list a few of these practical and technical challenges: (1) the protection and sensitivity of data, where data custodians and guardians prevent automatic repairing algorithms from changing the underlying data; (2) the heterogeneity of integrity constraints, which makes proposed techniques that tackle a single type of error inapplicable or ineffective in practice; (3) the lack of ground truth to validate repairing strategies; quality metrics such as minimal repairs have not been showing great results in practice; and (4) the lack of interactive tools for data quality that allow users and experts to reason about the problematic parts of the data and to explain the reasons behind these errors.**In this proposal, we focus on enabling data quality analytics and retrieval on large-scale inconsistent and dirty databases. The proposal pursues a set of research directions including (1) non-destructive data cleaning that represents and queries possible data repairs without changing the underlying data; (2) holistic data cleaning, which addresses the violations of multiple heterogeneous integrity constraints; (3) high-fidelity data repairing, which depends more on trusted data sources and experts, and depends less on heuristic quality metrics, such as minimal repairs; and (4) descriptive and prescriptive data quality analytics in practical dashboards that go beyond describing errors in the data to recommending ways to prevent future errors.**The proposed techniques will be implemented and tested in our previously developed system prototypes: UClean, a probabilistic and quality-aware database engine prototype, based on an open-source Database Management System; and NADEEF, an open source extensible data cleaning system. The goal is to build a generic framework that encapsulates efficient query processing algorithms to allow users to effectively query, analyze and explore large volumes of inconsistent and uncertain data.*The developed algorithms and dashboard will enable both the research community and industry to reason about the quality of available data sets, and to provide guidance on how to clean or enhance the quality of this data with respect to target applications or use cases.

对象跟踪、传感器网络、健康记录管理和Web数据集成等现代应用程序生成的数据包含不确定性，以及丢失值和重复等各种异常。虽然许多研究工作一直集中在数据清理和处理不一致的数据库上，但在真实环境中针对增加和从大型脏数据集中提取价值的各种技术和实践挑战进行的研究非常有限。列举几个实际和技术挑战：(1)数据的保护和敏感性，其中数据保管人和监护人阻止自动修复算法改变底层数据；(2)完整性约束的异构性，这使得解决单一类型错误的拟议技术在实践中不适用或无效；(3)缺乏基本事实来验证修复策略；最小修复等质量指标在实践中没有显示出很大的效果；以及(4)缺乏交互式数据质量工具，使用户和专家能够对数据中有问题的部分进行推理，并解释这些错误背后的原因。**在本提案中，我们侧重于在大规模不一致和肮脏的数据库上启用数据质量分析和检索。该方案追求一系列的研究方向，包括(1)无损数据清理，在不改变底层数据的情况下表示和查询可能的数据修复；(2)整体数据清理，解决多个异质完整性约束的违反；(3)高保真数据修复，它更依赖于可信的数据源和专家，而较少依赖启发式质量度量，如最小修复；以及(4)实用仪表板中的描述性和说明性数据质量分析，它不仅描述数据中的错误，还建议防止未来错误的方法。**所提出的技术将在我们之前开发的系统原型中实施和测试：UClean，一个基于开源数据库管理系统的概率和质量感知数据库引擎原型；以及NADEEF，一个开源的可扩展数据清理系统。其目标是建立一个封装高效查询处理算法的通用框架，以允许用户有效地查询、分析和探索大量不一致和不确定的数据。*开发的算法和仪表板将使研究界和行业能够对可用的数据集的质量进行推理，并就如何针对目标应用程序或用例来清理或提高这些数据的质量提供指导。