Characterizing and Improving Data Quality in Very Large Databases

数据挖掘和数据分析已经成为商业和政府组织的标准实践。 然而，分析结果仅与输入数据一样好。不幸的是，现代数据库和信息系统的规模和复杂性使其难以确保数据质量。特别是，对数据语义的理解不足（导致数据库设计不佳），数据随时间的演变（例如，由于整合了新的数据源），而且容易出错的收集可能导致数据不一致、不正确和不完整。 了解和监控数据质量以避免“垃圾输入-垃圾输出”问题是一个重要且具有挑战性的问题。 另一个挑战是，业务关键型应用程序经常收集连续的数据流，这些数据流必须“即时”处理，以支持实时决策。 最近提出了几种数据管理技术来处理流数据，包括数据流管理系统，流数据仓库和事件处理系统。 该建议为流数据质量的方法和工具建立了一个新的研究方向。 这包括描述流数据语义的模型，以及在新数据到达时增量检测和解决数据质量问题的算法。 提高数据流质量将使流处理系统更可用。 因此，所提出的研究对于可以通过真实的时间做出数据驱动的决策（例如，医疗保健、金融机构、电信公司、基于Web的公司（如Google、Yahoo！Facebook和Twitter、执法、智能电网管理和高速公路交通管理），以及数据库和信息系统供应商（例如，IBM、Oracle、Microsoft、Terrace、SAP、StreamBase Systems）。

Data mining and data analysis have become standard practice in business and government organizations. However, analysis results are only as good as the input data. Unfortunately, the size and complexity of modern databases and information systems make it difficult to ensure data quality. In particular, inadequate understanding of the data semantics (leading to poor database design), data evolution over time (e.g., due to integrating new data sources) and error-prone collection may lead to data that are inconsistent, incorrect and incomplete. Understanding and monitoring data quality to avoid the "garbage-in-garbage-out" problem is an important and challenging issue. An additional challenge is that business-critical applications often collect continuous streams of data, which must be processed "on the fly" to support real-time decision making. Several data management technologies have recently been proposed to handle streaming data, including data stream management systems, stream data warehouses and event-processing systems. This proposal establishes a new research direction into methodologies and tools for streaming data quality. This includes models that describe the semantics of streaming data, and algorithms that incrementally detect and resolve data quality issues as new data arrive. Improving data stream quality will make stream processing systems more usable. Thus, the proposed research is of interest to businesses and government organizations that can gain a competitive advantage by making data-driven decisions in real time (e.g., healthcare, financial institutions, telecommunications companies, Web-based companies such as Google, Yahoo!, Facebook and Twitter, law enforcement, smart power grid management, and highway traffic management), and to database and information systems vendors (e.g., IBM, Oracle, Microsoft, Teradata, SAP, StreamBase Systems).