Characterizing and Improving Data Quality in Very Large Databases

点击翻译按钮获取中文摘要

Data mining and data analysis have become standard practice in business and government organizations. However, analysis results are only as good as the input data. Unfortunately, the size and complexity of modern databases and information systems make it difficult to ensure data quality. In particular, inadequate understanding of the data semantics (leading to poor database design), data evolution over time (e.g., due to integrating new data sources) and error-prone collection may lead to data that are inconsistent, incorrect and incomplete. Understanding and monitoring data quality to avoid the "garbage-in-garbage-out" problem is an important and challenging issue. An additional challenge is that business-critical applications often collect continuous streams of data, which must be processed "on the fly" to support real-time decision making. Several data management technologies have recently been proposed to handle streaming data, including data stream management systems, stream data warehouses and event-processing systems. This proposal establishes a new research direction into methodologies and tools for streaming data quality. This includes models that describe the semantics of streaming data, and algorithms that incrementally detect and resolve data quality issues as new data arrive. Improving data stream quality will make stream processing systems more usable. Thus, the proposed research is of interest to businesses and government organizations that can gain a competitive advantage by making data-driven decisions in real time (e.g., healthcare, financial institutions, telecommunications companies, Web-based companies such as Google, Yahoo!, Facebook and Twitter, law enforcement, smart power grid management, and highway traffic management), and to database and information systems vendors (e.g., IBM, Oracle, Microsoft, Teradata, SAP, StreamBase Systems).