Uncertainty and Data Cleansing in the Stratosphere Cloud Data Management System

数据质量差的问题，表现为不确定、不完整和不一致的数据，渗透到数据管理中。支持各种应用程序和用例(包括文本提取和数据集成)的复杂数据管理系统，如Stratball，必须能够处理源数据和计算结果的内在不确定性和往往较差的质量。在平流层中，我们将开发检测和表示不确定数据、错误数据、不一致数据和不完整数据的方法。表示将不限于基本数据，更有趣的是，表示是转换、查询和其他计算的结果的数据。为此，我们在平流层内扩展了基本的代数运算。仅仅表示不确定性和劣质忽略了云环境的潜力。基于云的体系结构允许并要求重新思考不确定性管理和数据清理的基本概念：可伸缩、自适应和并行执行引擎提供的计算功能首次允许在查询上下文中进行临时数据清理，而不是在长时间手动创建的ETL过程中进行，并允许更广泛地探索充满不确定性的数据库的可能世界。不确定性不仅将被表示，而且将是可查询的；平流层的优化组件将允许不确定性约束的查询。糟糕的数据质量不仅会被表现出来，而且它的质量会得到提高；平流层将包括一组基本的操作符来清理数据，针对并行云环境进行优化，并集成到平流层的数据模型、查询语言和执行引擎中。

The problem of poor data quality, in the form of uncertain, incomplete, and inconsistent data, permeates data management. Complex data management systems, such as Stratosphere, that support various applications and use cases including text extraction and data integration must be able to handle the intrinsic uncertainty and often poor quality of source data and computed results. Within Stratosphere we will develop methods to detect and represent uncertain data, erroneous data, inconsistent data, and incomplete data. The representation will not be limited to base data, but, more interestingly, to data that is the result of transformations, queries, and other computations. To this end we extend the basic algebraic operations within Stratosphere. Merely representing uncertainty and poor quality ignores the potential of the Cloud environment. Stratosphere’s Cloud-based architecture allows and demands a rethinking of the basic notions of uncertainty management and data cleansing: The scalable, adaptive, and parallel execution engine provides compute capabilities that for the first time allow ad hoc data cleansing in the context of queries and not of long, manually created ETL processes and that allow a wider exploration of the possible worlds of uncertainty-laced databases. Uncertainty will not only be represented but will also be queryable; Stratosphere’s optimization component will allow uncertainty-constrained queries. Poor data quality will not only be represented, but its quality will be improved; Stratosphere will include a set of basic operators to cleanse data, optimized for the parallel Cloud environment, and integrated in Stratosphere’s data model, query language, and execution engine.