DataPrep: Human-in-the-Loop Data Preparation

DataPrep：人在环数据准备

• 批准号: RGPIN-2021-03995
• 负责人: Wang, Jiannan
• 金额: $ 3.5万
• 依托单位: Simon Fraser University
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2022
• 资助国家: 加拿大
• 起止时间: 2022-01-01 至 2023-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=750583
• 关键词: DataPrep; Human; Loop; Data; Preparation

Data preparation refers to the process of collecting, exploring, cleaning, transforming, and integrating data into a form for downstream analysis and modeling. It is widely regarded as the most time-consuming part in the data-science lifecycle. Although some efforts have been devoted to solving this problem, a survey released by Anaconda in 2020 shows that it is still the case that "Data preparation and cleansing takes valuable time away from real data science work and has a negative impact on overall job satisfaction". In recent years, Python has become the most popular programming language among data scientists. There are a wide range of Python libraries available to simplify different stages of the data science pipeline. Take Scikit-learn as an example. With its help, data scientists are able to spend much less time on the machine learning stage. Inspired by the great success of Scikit-learn, our proposed research program aims to build DataPrep (http://dataprep.ai), a human-in-the-loop data preparation system in Python. In the short term, we will work on three specific modules in DataPrep: i) DataPrep.EDA: a task-centric exploratory data analysis module; ii) DataPrep.Connector: a unified API wrapper to simplify web data collection; iii) DataPrep.Match: an automated model development module for entity matching. The long term goal is to build an all-in-one data preparation system that provides the easiest way for data scientists to prepare data in Python. Data preparation is a critical stage towards successful data science projects in many scientific fields. The market size of data preparation is estimated to be over 18 billion (USD) by 2027. Since our DataPrep system can greatly simplify data preparation, it will make a big impact on the field of data science from both academic and industrial perspectives.

数据准备是指收集、探索、清理、转换数据并将其集成到表单中以进行下游分析和建模的过程。它被广泛认为是数据科学生命周期中最耗时的部分。尽管已经为解决这一问题付出了一些努力，但Anaconda在2020年发布的一项调查显示，“数据准备和清洗占用了真实的数据科学工作的宝贵时间，并对整体工作满意度产生负面影响”的情况仍然存在。近年来，Python已经成为数据科学家中最流行的编程语言。有大量的Python库可用于简化数据科学管道的不同阶段。以Scikit-learn为例。在它的帮助下，数据科学家能够在机器学习阶段花费更少的时间。受到Scikit-learn巨大成功的启发，我们提议的研究计划旨在构建DataPrep（http：dataprep.ai），这是一个Python中的人在回路数据准备系统。 在短期内，我们将致力于DataPrep中的三个特定模块：i）DataPrep.EDA：以任务为中心的探索性数据分析模块; ii）DataPrep.Connector：简化Web数据收集的统一API包装器; iii）DataPrep.Match：用于实体匹配的自动化模型开发模块。长期目标是构建一个一体化的数据准备系统，为数据科学家提供用Python准备数据的最简单方法。数据准备是许多科学领域成功的数据科学项目的关键阶段。到2027年，数据准备的市场规模估计将超过180亿美元。由于我们的DataPrep系统可以大大简化数据准备，因此它将从学术和工业角度对数据科学领域产生重大影响。