A dynamic and user-centric data cleaning system for Watson Analytics

点击翻译按钮获取中文摘要

Poor data quality is a serious and costly problem affecting organizations across all industries. Real data is often dirty, containing missing, erroneous, incomplete, and duplicate values. It is estimated that poor data quality cost organizations between 15% and 25% of their operating budget. Existing data cleaning solutions focus on identifying inconsistencies that do not conform to prescribed data formats assuming the data remains relatively static. As modern applications move towards more dynamic search and analytics, new data quality solutions that support dynamic and adaptive data cleaning are needed.An increasing number of data analysis tools, such as Watson Analytics, provide flexible data browsing and querying abilities. In order to ensure reliable, trusted and relevant data analysis, dynamic data cleaning solutions that adapt to users' data analytic tasks are required. In particular, current data quality tools fail to adapt to: (1) individual user query preferences and specific data analytic tasks; (2) provide users with detailed data quality metrics that indicate the level of data cleanliness across different dimensions; and (3) utilize industry specific terminology, concepts, policies that can refine data quality recommendations for greater accuracy and relevance.In this project, we will develop a system for dynamic data cleaning that adapts to a user's analytical intentions and considers industry specific models for improved data quality.