DataPrep: Human-in-the-Loop Data Preparation

点击翻译按钮获取中文摘要

Data preparation refers to the process of collecting, exploring, cleaning, transforming, and integrating data into a form for downstream analysis and modeling. It is widely regarded as the most time-consuming part in the data-science lifecycle. Although some efforts have been devoted to solving this problem, a survey released by Anaconda in 2020 shows that it is still the case that "Data preparation and cleansing takes valuable time away from real data science work and has a negative impact on overall job satisfaction". In recent years, Python has become the most popular programming language among data scientists. There are a wide range of Python libraries available to simplify different stages of the data science pipeline. Take Scikit-learn as an example. With its help, data scientists are able to spend much less time on the machine learning stage. Inspired by the great success of Scikit-learn, our proposed research program aims to build DataPrep (http://dataprep.ai), a human-in-the-loop data preparation system in Python. In the short term, we will work on three specific modules in DataPrep: i) DataPrep.EDA: a task-centric exploratory data analysis module; ii) DataPrep.Connector: a unified API wrapper to simplify web data collection; iii) DataPrep.Match: an automated model development module for entity matching. The long term goal is to build an all-in-one data preparation system that provides the easiest way for data scientists to prepare data in Python. Data preparation is a critical stage towards successful data science projects in many scientific fields. The market size of data preparation is estimated to be over 18 billion (USD) by 2027. Since our DataPrep system can greatly simplify data preparation, it will make a big impact on the field of data science from both academic and industrial perspectives.