Scalable Cleaning, Integration and Analysis of Structured and Semi-Structured Inconsistent Data

所有垂直领域的企业(例如医疗保健、金融服务、制造商和保险公司)一直在积极地从各种来源收集数据，包括客户、交易、传感器和社交数据，以构建最终的数据资产。希望通过使用适当的分析技术，这些数据可以提供洞察力、方向和发现，从而提高客户满意度、实现更高的利润率，甚至激发新业务线的创建或实现新的发现。不幸的是，阻止这一美好愿景成为普遍现实的是数据本身；肮脏和孤立的数据是常态，而不是例外。因此，数据管理、清理和集成成为有效数据科学这一重大承诺的关键推动因素。《纽约时报》(2014年8月)的一篇文章指出，对于数据科学家来说，“清理”是获得洞察力的关键障碍。大规模数据清理以实现数据科学是这项提案的主要目标。数据清理通常由一系列活动描述，包括查找和修复异常和离群值、输入缺失值以及对代表同一实体的记录进行重复数据删除。主要目标是准备数据，以便通过各种工具进行挖掘和分析，以产生高质量的汇总和见解。管理和整合大量数据的任务提出了真正的理论和工程挑战。目前的大多数提案都存在根本性问题，这些问题阻碍了这些解决方案在实际工业和商业环境中的部署。我建议在数据质量方面进行基础研究，以得出可以在真实环境中部署的解决方案(新技术、新方法和新算法)。主要目标是能够对大规模不一致和肮脏的数据源进行有质量意识的分析和检索，释放数据科学的潜力。为了实现这一目标，我们打算调查的一些基本挑战包括：(1)开发可扩展到大型数据集的高效剖析和修复解决方案；(2)通过开发具有隐私意识的探索、错误检测和修复框架来解决敏感数据周围的隐私问题；(3)将数据清理建模为大规模统计推理问题，该问题考虑了所有可用的信号，包括业务规则、主数据和各种统计属性；(4)研究离群点检测问题的实用变体；以及(5)调查非结构化数据(例如文本)与结构化关系数据的集成中的质量问题，包括重新访问信息提取系统以包括质量约束。建议的技术将在多个开源系统原型中实现和测试，包括我们最新的基于机器学习的数据清理系统HoloClean。

Enterprises in all verticals (e.g., healthcare, financial services, manufacturers, and insurance companies) have been aggressively collecting data from a variety of sources including customers, transactions, sensors and social data to build the ultimate data asset. The hope is that by employing appropriate analysis techniques, this data can provide insights, directions, and findings that increase their customer satisfaction; achieve higher profit margins; or even inspire the creation of new lines of business or enable new discoveries. Unfortunately, what prevents this fine vision from being a pervasive reality is the data itself; dirty and siloed data is the norm rather than the exception. Consequently, data curation, cleaning and integration become key enablers to the big promise of effective data science. An article in the New York Times (August of 2014) indicated that for data scientists, "cleaning" is key hurdle to insights. Large scale data cleaning to enable data science is the main goal of this proposal. Data cleaning is often described by a set of activities including finding and fixing anomalies and outliers, imputing missing values, and deduplicating records representing the same entity. The main objective is to prepare data to be mined and analyzed by a variety of tools to produce high quality aggregates and insights. The task of curating and integrating large amounts of data presents real theoretical and engineering challenges. Most current proposals suffer from fundamental problems that hinder any of these solutions from being deployed in practical industry and business settings. I propose to conduct fundamental research in data quality leading to solutions (new technologies, methods and algorithms) that can be deployed in real environments. The main objective is to enable quality-aware analytics on and retrieval from large-scale inconsistent and dirty data sources, unleashing the potential of data science. Some of the fundamental challenges in achieving this objective, which we intend to investigate, include: (1) developing efficient profiling and repair solutions that scale to large data sets; (2) addressing the privacy concerns around sensitive data by developing privacy-aware exploration, error detection, and repair framework; (3) modelling data cleaning as large scale statistical inference problem that takes into account all available signals including business rules, master data and various statistical properties; (4) studying practical variants of the outlier detection problem; and (5) investigate the quality issues in integrating unstructured data (such as text), with structured relational data, including revisiting information extraction systems to include quality constraints. The proposed techniques will be implemented and tested in multiple open-source system prototypes, including HoloClean, our recent system for machine learning-based data cleaning.