A dynamic and user-centric data cleaning system for Watson Analytics

数据质量差是影响所有行业组织的一个严重且代价高昂的问题。真实数据通常是脏的，包含丢失的、错误的、不完整的和重复的值。据估计，糟糕的数据质量使组织损失了15%到25%的运营预算。现有的数据清理解决方案侧重于识别不符合规定数据格式的不一致性，假设数据保持相对静态。随着现代应用程序转向更动态的搜索和分析，需要支持动态和自适应数据清理的新数据质量解决方案。****越来越多的数据分析工具（如Watson Analytics）提供了灵活的数据浏览和查询功能。为了保证数据分析的可靠性、可信度和相关性，需要根据用户的数据分析任务提供动态的数据清洗解决方案。特别是，目前的数据质量工具无法适应：(1)个人用户查询偏好和特定的数据分析任务；(2)为用户提供详细的数据质量指标，表明不同维度的数据清洁水平；(3)利用行业特定的术语、概念和策略来改进数据质量建议，以提高准确性和相关性。****在这个项目中，我们将开发一个动态数据清理系统，该系统适应用户的分析意图，并考虑行业特定模型以提高数据质量。* * * *

Poor data quality is a serious and costly problem affecting organizations across all industries. Real data is often dirty, containing missing, erroneous, incomplete, and duplicate values. It is estimated that poor data quality cost organizations between 15% and 25% of their operating budget. Existing data cleaning solutions focus on identifying inconsistencies that do not conform to prescribed data formats assuming the data remains relatively static. As modern applications move towards more dynamic search and analytics, new data quality solutions that support dynamic and adaptive data cleaning are needed.****An increasing number of data analysis tools, such as Watson Analytics, provide flexible data browsing and querying abilities. In order to ensure reliable, trusted and relevant data analysis, dynamic data cleaning solutions that adapt to users' data analytic tasks are required. In particular, current data quality tools fail to adapt to: (1) individual user query preferences and specific data analytic tasks; (2) provide users with detailed data quality metrics that indicate the level of data cleanliness across different dimensions; and (3) utilize industry specific terminology, concepts, policies that can refine data quality recommendations for greater accuracy and relevance.****In this project, we will develop a system for dynamic data cleaning that adapts to a user's analytical intentions and considers industry specific models for improved data quality. ****