A dynamic and user-centric data cleaning system for Watson Analytics

Watson Analytics 的动态且以用户为中心的数据清理系统

• 批准号: 490788-2015
• 负责人: Chiang, FeiYen
• 金额: $ 4.95万
• 依托单位: McMaster University
• 依托单位国家: 加拿大
• 项目类别: Collaborative Research and Development Grants
• 财政年份: 2017
• 资助国家: 加拿大
• 起止时间: 2017-01-01 至 2018-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=618852
• 关键词: dynamic; user; centric; data; cleaning

Poor data quality is a serious and costly problem affecting organizations across all industries. Real data is often dirty, containing missing, erroneous, incomplete, and duplicate values. It is estimated that poor data quality cost organizations between 15% and 25% of their operating budget. Existing data cleaning solutions focus on identifying inconsistencies that do not conform to prescribed data formats assuming the data remains relatively static. As modern applications move towards more dynamic search and analytics, new data quality solutions that support dynamic and adaptive data cleaning are needed.An increasing number of data analysis tools, such as Watson Analytics, provide flexible data browsing and querying abilities. In order to ensure reliable, trusted and relevant data analysis, dynamic data cleaning solutions that adapt to users' data analytic tasks are required. In particular, current data quality tools fail to adapt to: (1) individual user query preferences and specific data analytic tasks; (2) provide users with detailed data quality metrics that indicate the level of data cleanliness across different dimensions; and (3) utilize industry specific terminology, concepts, policies that can refine data quality recommendations for greater accuracy and relevance.In this project, we will develop a system for dynamic data cleaning that adapts to a user's analytical intentions and considers industry specific models for improved data quality.

数据质量差是一个严重且代价高昂的问题，影响着所有行业的组织。真实数据通常是脏的，包含缺失、错误、不完整和重复的值。据估计，糟糕的数据质量会让组织损失 15% 到 25% 的运营预算。现有的数据清理解决方案侧重于识别不符合规定数据格式的不一致情况（假设数据保持相对静态）。随着现代应用程序转向更加动态的搜索和分析，需要支持动态和自适应数据清理的新数据质量解决方案。越来越多的数据分析工具（例如 Watson Analytics）提供灵活的数据浏览和查询能力。为了确保可靠、可信和相关的数据分析，需要适应用户数据分析任务的动态数据清理解决方案。特别是，当前的数据质量工具无法适应：（1）个人用户查询偏好和特定的数据分析任务； （2）为用户提供详细的数据质量指标，表明不同维度的数据清洁程度； (3) 利用行业特定的术语、概念和政策来完善数据质量建议，以提高准确性和相关性。在这个项目中，我们将开发一个动态数据清理系统，适应用户的分析意图，并考虑行业特定模型以提高数据质量。