POSE: Phase II: Expanding the data.table ecosystem for efficient big data manipulation in R

来自北亚利桑那大学的Toby Hocking教授得到了技术、创新和伙伴关系管理局(TIP)开放源代码生态系统(POST)计划颁发的奖项的支持。系统地收集和记录的数据比人类历史上任何时候都多，需要高效的软件包来存储和分析这些数据，使用有限的计算资源。一个领先的、最先进的例子是data.table，它是用于内存中数据操作/分析的免费/开源软件，实现为一个带有C代码的R包，在计算时间和内存使用方面都非常高效。尽管data.table有相当数量的用户(包括数千个从data.table导入功能的其他R包)，但data.table的增长受到以下因素的限制：(1)其扁平/非正式的领导结构，顶层只有一位作者可以批准新的代码贡献；(2)缺乏文档/翻译和促进多样性/包容性的社区标准；以及(3)缺乏系统软件测试的基础设施。通过解决这些问题，该项目将扩大由data.table的用户、贡献者和开发人员组成的开源生态系统。特别是，该项目将创建(1)具有新的层级领导结构的书面治理文件，(2)为新用户/贡献者提供新的文件材料，包括鼓励多样性/包容性的翻译和社区标准，以及(3)新的测试软件和基础设施。此外，该项目还计划通过衡量重要指标的变化(独特贡献者的数量、贡献者的多样性、依赖包的数量等)，系统地评估这项工作对数据和表格生态系统的更广泛影响。这个项目的结果将是一个自我维持的数据开源生态系统。表，这将使它成长为一个更强大的数据分析工具，在未来将被更多的人和更多样化的人使用。这个奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Professor Toby Hocking from Northern Arizona University is supported by an award from the Pathways to Enable Open-Source Ecosystems (POSE) program in the Directorate for Technology, Innovation and Partnerships (TIP). More data is being systematically gathered and recorded than at any previous time in human history, and efficient software packages are required to store and analyze these data using limited computational resources. A leading, state-of-the-art example is data.table, which is free/open-source software for in-memory data manipulation/analysis, implemented as an R package with C code that is highly efficient in terms of both computation time and memory usage. Although data.table has a substantial number of users (including thousands of other R packages which import functionality from data.table), the growth of data.table is limited by (1) its flat/informal leadership structure with only one author at the top who can approve new code contributions, (2) lack of documentation/translations and community standards for promoting diversity/inclusion, and (3) lack of infrastructure for systematic software testing. This project will expand the open-source ecosystem of users, contributors, and developers of data.table, by addressing these issues. In particular, this project will create (1) a written governance document with a new hierarchical leadership structure, (2) new documentation materials for onboarding new users/contributors, including translations and community standards to encourage diversity/inclusion, and (3) new testing software and infrastructure. Furthermore, the project includes plans to systematically evaluate the broader impacts of this effort on the data.table ecosystem by measuring changes to important metrics (number of unique contributors, diversity of contributors, number of dependent packages, etc). The result of this project will be a self-sustaining open-source ecosystem for data.table, which will allow it to grow into a more powerful data analysis tool that will be used by more people, and more diverse people, in the future.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.