Big Data Cleaning

点击翻译按钮获取中文摘要

Organizations are finding it increasingly difficult to reap value from their data due to poor data quality. Poor data quality is a barrier to effective and high quality decision-making based on data. Integrity constraints (business rules) are the fundamental tools used to preserve data integrity. They specify the domain semantics that should hold over the data. Declarative data cleaning has emerged as an effective tool for both assessing and improving the quality of data. We will address some important challenges, in applying declarative data cleaning to big data, that arise due to the scale, complexity, and massive heterogeneity of such data. Given the proliferation of different data formats and data representations that exist in modern applications, the integration of these heterogeneous data sources leads to subtle inconsistencies that are not handled by the existing methods. First, given the dynamic nature of big data, we will develop new continuous data cleaning methods for dynamic data environments. As the data and constraints evolve we need to identify repairs based on this incremental changes, without having to start the repair process from scratch each time. Second, we will investigate metrical and ontological constrains to enhance declarative data cleaning and explore holistic data cleaning with an extensible rule specification. We will develop a novel framework that combines statistical (employing principles of statistics) and logical (using various forms of logical reasoning over declarative dependencies) data cleaning. Recent work in data cleaning has proposed solutions largely isolated to only one of these areas. Third, recognizing the massive heterogeneity of big data and that automation will rarely provide 100% accuracy, we will develop new techniques to explain the provenance (lineage) of data cleaning solutions. Provenance helps users to understand why (and how) a cleaning decision was derived and can enable users to debug and correct automated solutions. We will also use data provenance to guide a cleaning algorithm to choose more accurate repairs. The proposed research is beneficial for Canadian (and international) business and government organizations. Organizations can significantly benefit by making analytical decisions over high quality data. Our solutions can be utilized in healthcare, telecommunication (e.g., Rogers and AT&T), financial (e.g., the Bank of Montreal and TD Bank) and governmental (e.g., Statistics Canada and the Ministry of Transportation) institutions. The outcomes of this research will also be of interest to software vendors, such as IBM, Oracle, SAP and Microsoft. The proposed program will train students in data management systems and place them in a competitive position while applying for jobs in academia and industry. We expect up to ten students (including undergraduate students) to receive training in this research program.