CAREER: Data Quality Management through Statistical Quality Control and Data Mining

随着海量数据的大量涌现，数据质量问题显得尤为重要。许多商业和工业应用程序严重依赖于存储在不同数据库和数据仓库中的信息质量。本研究项目的目标是开发一种系统的数据质量分析和改进方法，以实现在不完全信息环境下的稳健决策。该项目为数据质量评估和评价制定了一个统一的框架，提供了通过信息生产和管理改进数据质量的实际解决方案，并通过维护一个网站来传播研究成果，以提高学术界和工业界专业人员对信息质量的认识。该方法包括建立贝叶斯网络模型以捕捉数据质量度量之间的相互关系，应用统计抽样方案和数据挖掘方法进行数据质量评估，以及推广用于根本原因识别和数据质量改进的统计技术。使用来自电信和信息技术(IT)行业的合成示例和真实案例对这些技术进行评估和验证。预计该项目的成果将是一般性的，并提供可应用于不同数据密集型应用程序的数据质量管理的具体基础。该项目将对信息质量管理的先进理论和方法、加强决策以及建立一支数据质量保证研究人员和从业人员队伍产生广泛影响。从这一研究项目中获得的知识和成果将通过互联网(http://www.stevens.edu/engineering/seem/Research/projects/DataQuality.html)，在会议、讲习班和各级课程中广泛传播。

Data quality problem is of great importance due to the emergence of large volumes of data. Many business and industrial applications critically rely on the quality of information stored in diverse databases and data warehouses. The goal of this research project is to develop a systematic methodology of data quality analysis and improvement to achieve robust decision making under imperfect information environments. The project develops a unified framework for data quality assessment and evaluation, deliveries practical solutions to improve data quality through information production and management, and disseminates research findings by maintaining a website to increase the awareness of information quality among academic and industrial professionals. The approach consists of developing Bayesian network models to capture inter-relationships between data quality metrics, applying statistical sampling schemes and data mining methods for data quality assessment, and generalizing statistical techniques for root cause identification and data quality improvement. The techniques are evaluated and validated using synthetic examples and real-life cases from telecommunication and information technologies (IT) industries. The outcomes of the project are expected to be generic and provide a concrete basis of data quality management that can be applied to different data-intensive applications. The project will have broad impacts on advanced theory and methodology of information quality management, enhanced decision making, and the creation of a workforce of data quality assurance researchers and practitioners. Knowledge gained and results obtained from this research project will be broadly disseminated via Internet (http://www.stevens.edu/engineering/seem/Research/projects/DataQuality.html), in conferences, workshops, and various levels of courses.