A dynamic and user-centric data cleaning system for Watson Analytics

数据质量差是一个严重且代价高昂的问题，影响着所有行业的组织。真实的数据通常是脏的，包含缺失、错误、不完整和重复的值。 据估计，较差的数据质量使组织损失了其运营预算的15%到25%。 现有的数据清理解决方案集中于识别不符合规定的数据格式的不一致性，假设数据保持相对静态。 随着现代应用程序向更动态的搜索和分析方向发展，需要支持动态和自适应数据清理的新数据质量解决方案。越来越多的数据分析工具，如沃森分析，提供灵活的数据浏览和查询能力。 为了确保可靠、可信和相关的数据分析，需要适应用户数据分析任务的动态数据清洗解决方案。 特别是，当前的数据质量工具无法适应：（1）个人用户查询偏好和特定的数据分析任务;（2）为用户提供详细的数据质量指标，指示不同维度的数据清洁度水平;以及（3）利用行业特定的术语，概念，策略，可以改进数据质量建议，以提高准确性和相关性。在这个项目中，我们将开发一个动态数据清理系统，以适应用户的分析意图，并考虑行业特定的模型，以提高数据质量。****

Poor data quality is a serious and costly problem affecting organizations across all industries. Real data is often dirty, containing missing, erroneous, incomplete, and duplicate values. It is estimated that poor data quality cost organizations between 15% and 25% of their operating budget. Existing data cleaning solutions focus on identifying inconsistencies that do not conform to prescribed data formats assuming the data remains relatively static. As modern applications move towards more dynamic search and analytics, new data quality solutions that support dynamic and adaptive data cleaning are needed.****An increasing number of data analysis tools, such as Watson Analytics, provide flexible data browsing and querying abilities. In order to ensure reliable, trusted and relevant data analysis, dynamic data cleaning solutions that adapt to users' data analytic tasks are required. In particular, current data quality tools fail to adapt to: (1) individual user query preferences and specific data analytic tasks; (2) provide users with detailed data quality metrics that indicate the level of data cleanliness across different dimensions; and (3) utilize industry specific terminology, concepts, policies that can refine data quality recommendations for greater accuracy and relevance.****In this project, we will develop a system for dynamic data cleaning that adapts to a user's analytical intentions and considers industry specific models for improved data quality. ****