DataPrep: Human-in-the-Loop Data Preparation

数据准备是指收集、探索、清理、转换数据并将其集成到一种形式中以供下游分析和建模的过程。它被广泛认为是数据科学生命周期中最耗时的部分。尽管已经投入了一些努力来解决这个问题，但蟒蛇在2020年发布的一项调查显示，仍然存在这样的情况，即数据准备和清理占用了宝贵的时间，离开了真正的数据科学工作，并对整体工作满意度产生了负面影响。近年来，Python已经成为数据科学家中最受欢迎的编程语言。有许多可用来简化数据科学管道的不同阶段的Python库。以Scikit-Learn为例。在它的帮助下，数据科学家能够在机器学习阶段花费更少的时间。受到Scikit-Learn巨大成功的启发，我们提出的研究计划旨在用Python语言构建DataPrep(http://dataprep.ai)，)一个人在循环中的数据准备系统。短期内，我们将致力于DataPrep中的三个特定模块：i)DataPrep.EDA：一个以任务为中心的探索性数据分析模块；ii)DataPrep.Connector：一个统一的API包装器，用于简化Web数据收集；iii)DataPrep.Match：一个用于实体匹配的自动化模型开发模块。我们的长期目标是建立一个一体化的数据准备系统，为数据科学家提供最简单的方法来用Python准备数据。数据准备是许多科学领域数据科学项目取得成功的关键阶段。预计到2027年，数据准备的市场规模将超过180亿美元。由于我们的DataPrep系统可以极大地简化数据准备，它将从学术和产业角度对数据科学领域产生重大影响。

Data preparation refers to the process of collecting, exploring, cleaning, transforming, and integrating data into a form for downstream analysis and modeling. It is widely regarded as the most time-consuming part in the data-science lifecycle. Although some efforts have been devoted to solving this problem, a survey released by Anaconda in 2020 shows that it is still the case that "Data preparation and cleansing takes valuable time away from real data science work and has a negative impact on overall job satisfaction". In recent years, Python has become the most popular programming language among data scientists. There are a wide range of Python libraries available to simplify different stages of the data science pipeline. Take Scikit-learn as an example. With its help, data scientists are able to spend much less time on the machine learning stage. Inspired by the great success of Scikit-learn, our proposed research program aims to build DataPrep (http://dataprep.ai), a human-in-the-loop data preparation system in Python. In the short term, we will work on three specific modules in DataPrep: i) DataPrep.EDA: a task-centric exploratory data analysis module; ii) DataPrep.Connector: a unified API wrapper to simplify web data collection; iii) DataPrep.Match: an automated model development module for entity matching. The long term goal is to build an all-in-one data preparation system that provides the easiest way for data scientists to prepare data in Python. Data preparation is a critical stage towards successful data science projects in many scientific fields. The market size of data preparation is estimated to be over 18 billion (USD) by 2027. Since our DataPrep system can greatly simplify data preparation, it will make a big impact on the field of data science from both academic and industrial perspectives.