Towards Scalable Dataframe Systems
Towards Scalable Dataframe Systems
复制标题
DOI：
10.14778/3407790.3407807
复制
发表时间：
2020-07-01
影响因子：
2.5
通讯作者：
Parameswaran, Aditya
中科院分区：
文献类型：
--
作者：
Petersohn, Devin;Macke, Stephen;Parameswaran, Aditya
数据框架是一种流行的抽象，用于表示，准备和分析数据。尽管在R和Python中的嵌套框架库取得了显著的成功，但即使在中等规模的数据集上，嵌套框架也面临着性能问题。此外，有显着的模糊性，关于框架语义。在本文中，我们列出了可扩展数据框系统的愿景和路线图。为了展示这一领域的潜力，我们报告了我们构建MODIN的经验，MODIN是当今使用最广泛和最复杂的框架API的扩展实现，Python的pandas。以pandas为参考，我们提出了一个简单的数据模型和代数，用于在该领域进行讨论。在此基础上，我们制定了一个开放式研究机会的议程，其中框架的独特功能将需要在数据管理的许多方面扩展最先进的技术。我们讨论了签名数据框架功能的影响，包括灵活的模式，排序，行/列等价性，数据/元数据的流动性，以及零碎的，尝试和错误为基础的方法来与框架进行交互。
Dataframes are a popular abstraction to represent, prepare, and analyze data. Despite the remarkable success of dataframe libraries in R and Python, dataframes face performance issues even on moderately large datasets. Moreover, there is significant ambiguity regarding dataframe semantics. In this paper we lay out a vision and roadmap for scalable dataframe systems. To demonstrate the potential in this area, we report on our experience building MODIN, a scaled-up implementation of the most widely-used and complex dataframe API today, Python's pandas. With pandas as a reference, we propose a simple data model and algebra for dataframes to ground discussion in the field. Given this foundation, we lay out an agenda of open research opportunities where the distinct features of dataframes will require extending the state of the art in many dimensions of data management. We discuss the implications of signature data-frame features including flexible schemas, ordering, row/column equivalence, and data/metadata fluidity, as well as the piecemeal, trial-and-error-based approach to interacting with dataframes.