C 2 Metadata: Automating the Capture of Data Transformations from Statistical Scripts in Data Documentation
C 2 Metadata: Automating the Capture of Data Transformations from Statistical Scripts in Data Documentation
复制标题
C 2 元数据：从数据文档中的统计脚本自动捕获数据转换
DOI：
10.1145/3299869.3320241
复制
发表时间：
2019
期刊：
影响因子：
--
通讯作者：
Jagadish, H. V.
中科院分区：
文献类型：
--
作者：
Song, Jie;Alter, George;Jagadish, H. V.
数据集通常是通过使用统计软件包处理原始数据而得出的。必须根据原始输入和对其应用的操作来记录数据集的派生。统计资料包通常在记录由此产生的派生数据的出处方面提供的帮助有限。充其量，统计包执行的操作在脚本中描述。完全不同的表示使用户很难理解这些脚本。为了应对这些挑战，我们创建了连续元数据捕获(C2Metadata)，这是一个在统计包的脚本中捕获数据转换并以易于理解的标准格式将其表示为元数据的系统。为此，我们设计了一个结构化数据转换代数(SDTA)，它使用一小部分代数运算符来表示实践中执行的大部分数据操作。然后，我们在关系代数的启发下，用一种称为SDTL的数据转换规范语言实现SDTA。在本演示中，我们展示了C2METADATA从至少两种语言的示例转换脚本池中捕获数据转换：SPSS和Stata(SAS和R正在开发中)，用于大型学术存储库中的社会科学数据。我们将允许观众使用基于Web的界面来探索C2Metadata，可视化中间步骤，并跟踪不同级别数据的来源和变化，以更好地了解该过程。
Datasets are often derived by manipulating raw data with statistical software packages. The derivation of a dataset must be recorded in terms of both the raw input and the manipulations applied to it. Statistics packages typically provide limited help in documenting provenance for the resulting derived data. At best, the operations performed by the statistical package are described in a script. Disparate representations make these scripts hard to understand for users. To address these challenges, we created Continuous Capture of Metadata (C2Metadata), a system to capture data transformations in scripts for statistical packages and represent it as metadata in a standard format that is easy to understand. We do so by devising a Structured Data Transformation Algebra (SDTA), which uses a small set of algebraic operators to express a large fraction of data manipulation performed in practce. We then implement SDTA, inspired by relational algebra, in a data transformation specification language we call SDTL. In this demonstration, we showcase C2metadata's capture of data transformations from a pool of sample transformation scripts in at least two languages: SPSS and Stata (SAS and R are under development), for social science data in a large academic repository. We will allow the audience to explore C2Metadata using a web-based interface, visualize the intermediate steps and trace the provenance and changes of data at different levels for better understanding of the process.