Cleaning and Analysis of Large Uncertain and Inconsistent Data Sources

点击翻译按钮获取中文摘要

Data generated by modern applications such as object tracking, sensor networks, health record management, and Web data integration involves uncertainty, and various anomalies such as missing values and duplication. While many research efforts have been focusing on data cleaning and dealing with inconsistent databases, very limited research has been adopted in real settings for various technical and practical challenges around increasing and extracting value from large dirty data sets. To list a few of these practical and technical challenges: (1) the protection and sensitivity of data, where data custodians and guardians prevent automatic repairing algorithms from changing the underlying data; (2) the heterogeneity of integrity constraints, which makes proposed techniques that tackle a single type of error inapplicable or ineffective in practice; (3) the lack of ground truth to validate repairing strategies; quality metrics such as minimal repairs have not been showing great results in practice; and (4) the lack of interactive tools for data quality that allow users and experts to reason about the problematic parts of the data and to explain the reasons behind these errors.In this proposal, we focus on enabling data quality analytics and retrieval on large-scale inconsistent and dirty databases. The proposal pursues a set of research directions including (1) non-destructive data cleaning that represents and queries possible data repairs without changing the underlying data; (2) holistic data cleaning, which addresses the violations of multiple heterogeneous integrity constraints; (3) high-fidelity data repairing, which depends more on trusted data sources and experts, and depends less on heuristic quality metrics, such as minimal repairs; and (4) descriptive and prescriptive data quality analytics in practical dashboards that go beyond describing errors in the data to recommending ways to prevent future errors.The proposed techniques will be implemented and tested in our previously developed system prototypes: UClean, a probabilistic and quality-aware database engine prototype, based on an open-source Database Management System; and NADEEF, an open source extensible data cleaning system. The goal is to build a generic framework that encapsulates efficient query processing algorithms to allow users to effectively query, analyze and explore large volumes of inconsistent and uncertain data.The developed algorithms and dashboard will enable both the research community and industry to reason about the quality of available data sets, and to provide guidance on how to clean or enhance the quality of this data with respect to target applications or use cases.