Characterizing and Improving Data Quality in Very Large Databases

数据挖掘和数据分析已经成为企业和政府机构的标准做法。然而，分析结果只与输入数据一样好。遗憾的是，现代数据库和信息系统的规模和复杂性使其难以保证数据质量。特别是，对数据语义的不充分理解(导致糟糕的数据库设计)、数据随时间的演变(例如，由于集成新的数据源)以及容易出错的收集可能导致数据不一致、不正确和不完整。了解和监测数据质量，避免垃圾进垃圾出问题，是一个重要而具有挑战性的问题。 另一个挑战是，业务关键型应用程序通常会收集连续的数据流，这些数据流必须在运行时进行处理，以支持实时决策。最近提出了几种数据管理技术来处理流数据，包括数据流管理系统、流数据仓库和事件处理系统。这一建议建立了对流数据质量的方法和工具的新的研究方向。这包括描述流数据语义的模型，以及在新数据到达时增量检测和解决数据质量问题的算法。 数据流质量的提高将使流处理系统更易于使用。因此，建议的研究对能够通过实时做出数据驱动的决策来获得竞争优势的企业和政府组织(例如，医疗保健、金融机构、电信公司、基于Web的公司，例如谷歌、雅虎、Facebook和Twitter、执法、智能电网管理和高速公路交通管理)以及数据库和信息系统供应商(例如，IBM、甲骨文、微软、Teradata、SAP、Streambase Systems)感兴趣。

Data mining and data analysis have become standard practice in business and government organizations. However, analysis results are only as good as the input data. Unfortunately, the size and complexity of modern databases and information systems make it difficult to ensure data quality. In particular, inadequate understanding of the data semantics (leading to poor database design), data evolution over time (e.g., due to integrating new data sources) and error-prone collection may lead to data that are inconsistent, incorrect and incomplete. Understanding and monitoring data quality to avoid the "garbage-in-garbage-out" problem is an important and challenging issue. An additional challenge is that business-critical applications often collect continuous streams of data, which must be processed "on the fly" to support real-time decision making. Several data management technologies have recently been proposed to handle streaming data, including data stream management systems, stream data warehouses and event-processing systems. This proposal establishes a new research direction into methodologies and tools for streaming data quality. This includes models that describe the semantics of streaming data, and algorithms that incrementally detect and resolve data quality issues as new data arrive. Improving data stream quality will make stream processing systems more usable. Thus, the proposed research is of interest to businesses and government organizations that can gain a competitive advantage by making data-driven decisions in real time (e.g., healthcare, financial institutions, telecommunications companies, Web-based companies such as Google, Yahoo!, Facebook and Twitter, law enforcement, smart power grid management, and highway traffic management), and to database and information systems vendors (e.g., IBM, Oracle, Microsoft, Teradata, SAP, StreamBase Systems).