Scalable Cleaning, Integration and Analysis of Structured and Semi-Structured Inconsistent Data

所有垂直领域的企业（例如，医疗保健、金融服务、制造商和保险公司）一直在积极地从各种来源收集数据，包括客户、交易、传感器和社交数据，以构建最终的数据资产。希望通过采用适当的分析技术，这些数据可以提供洞察力，方向和发现，提高客户满意度;实现更高的利润率;甚至激励创建新的业务线或实现新的发现。不幸的是，阻止这种美好愿景成为普遍现实的是数据本身;肮脏和孤立的数据是常态而不是例外。因此，数据管理、清理和集成成为实现有效数据科学的关键因素。纽约时报（2014年8月）的一篇文章指出，对于数据科学家来说，“清理”是获得洞察力的关键障碍。大规模数据清洗以实现数据科学是该提案的主要目标。 数据清理通常由一组活动来描述，包括查找和修复异常和离群值，估算缺失值以及删除代表同一实体的重复记录。主要目标是准备数据，通过各种工具进行挖掘和分析，以产生高质量的聚合和见解。管理和整合大量数据的任务提出了真实的理论和工程挑战。大多数当前的建议都存在根本性的问题，这些问题阻碍了这些解决方案中的任何一个在实际的工业和商业环境中部署。 我建议在数据质量方面进行基础研究，以获得可以在真实的环境中部署的解决方案（新技术，方法和算法）。其主要目标是实现对大规模不一致和脏数据源的质量感知分析和检索，释放数据科学的潜力。实现这一目标的一些基本挑战，我们打算调查，包括：（1）开发有效的分析和修复解决方案，可扩展到大型数据集;（2）通过开发隐私感知的探索，错误检测和修复框架来解决敏感数据的隐私问题;（3）将数据清理建模为考虑所有可用信号（包括业务规则、主数据和各种统计属性）的大规模统计推断问题;（4）研究异常值检测问题的实际变体;（5）研究将非结构化数据（如文本）与结构化关系数据集成时的质量问题，包括重新访问信息提取系统以包括质量约束。 所提出的技术将在多个开源系统原型中实现和测试，包括HoloClean，我们最近的基于机器学习的数据清理系统。

Enterprises in all verticals (e.g., healthcare, financial services, manufacturers, and insurance companies) have been aggressively collecting data from a variety of sources including customers, transactions, sensors and social data to build the ultimate data asset. The hope is that by employing appropriate analysis techniques, this data can provide insights, directions, and findings that increase their customer satisfaction; achieve higher profit margins; or even inspire the creation of new lines of business or enable new discoveries. Unfortunately, what prevents this fine vision from being a pervasive reality is the data itself; dirty and siloed data is the norm rather than the exception. Consequently, data curation, cleaning and integration become key enablers to the big promise of effective data science. An article in the New York Times (August of 2014) indicated that for data scientists, "cleaning" is key hurdle to insights. Large scale data cleaning to enable data science is the main goal of this proposal. Data cleaning is often described by a set of activities including finding and fixing anomalies and outliers, imputing missing values, and deduplicating records representing the same entity. The main objective is to prepare data to be mined and analyzed by a variety of tools to produce high quality aggregates and insights. The task of curating and integrating large amounts of data presents real theoretical and engineering challenges. Most current proposals suffer from fundamental problems that hinder any of these solutions from being deployed in practical industry and business settings. I propose to conduct fundamental research in data quality leading to solutions (new technologies, methods and algorithms) that can be deployed in real environments. The main objective is to enable quality-aware analytics on and retrieval from large-scale inconsistent and dirty data sources, unleashing the potential of data science. Some of the fundamental challenges in achieving this objective, which we intend to investigate, include: (1) developing efficient profiling and repair solutions that scale to large data sets; (2) addressing the privacy concerns around sensitive data by developing privacy-aware exploration, error detection, and repair framework; (3) modelling data cleaning as large scale statistical inference problem that takes into account all available signals including business rules, master data and various statistical properties; (4) studying practical variants of the outlier detection problem; and (5) investigate the quality issues in integrating unstructured data (such as text), with structured relational data, including revisiting information extraction systems to include quality constraints. The proposed techniques will be implemented and tested in multiple open-source system prototypes, including HoloClean, our recent system for machine learning-based data cleaning.