Framework: Awkward Arrays - Accelerating scientific data analysis on irregularly shaped data

如今，科学家不仅要成为各自领域的专家，还要成为程序员。任何简化数据分析计算体操的软件都是受欢迎的，因为它允许科学家将更多的注意力集中在他们想要计算的内容上，而不是如何计算，但易用性和计算速度之间通常存在紧张关系。更快的计算意味着分析更多的数据，但无错误的代码最重要。Awkward Array是一个软件库，它对不适合整齐行的数据执行计算。例如，一队海底探测器可能分别测量不同深度的海洋温度，但Excel、Pandas和SQL等数据分析工具要求将测量结果排列在每行列数相同的矩形表中。当可变数量的测量值嵌套在可变数量的实体中时，问题更加严重。传统上，科学家们要么使用缓慢的脚本语言，要么使用无情的“裸金属”语言来对这些非表格数据集执行计算。Awkward Array概括了数组概念，因此曾经仅适用于矩形表的简单快速表达式现在适用于不规则形状的数据，允许高级脚本语言中的裸金属速度。该项目将Awkward Array的适用性扩展到了它最初用于粒子物理学的问题领域之外，扩展到了各种科学领域，包括海洋学，天文学，遗传学，化学和医疗保健。它还将Awkward Array库与流行的数据科学和机器学习工具集成在一起，并将实现扩展到GPU，以极快地处理相同的数组习惯用法。科学家使用基于NumPy的工具（如Pandas）来有效地分析大型和规则形状的数据。将数字表打包到连续的数组中可以使操作进行预编译和快速;将操作表示为带有隐式循环的简洁命令可以使它们在数据探索期间更容易阅读和更快地输入。然而，科学数据往往具有复杂性，不适合表格格式。这迫使科学家编写带有显式循环的程序，这在Python中很慢。但是，如何分析大型和不规则形状的数据呢？Awkward Array是NumPy的一个推广;它是一个Python库，定义了具有任意类型的对象数组和一套通用操作。与NumPy数组一样，Awkward Arrays被打包到连续的缓冲区中以提高效率，但与NumPy数组不同的是，它们可以包括可变长度列表，嵌套字段，缺失值和混合类型。Awward Arrays使用的内存比等价的Python对象少10倍，计算速度快100倍。该项目将Awkward Array概括为科学的基础库。这是一项跨学科的工作，扩展了Awkward Array，以有效地解决科学家在各个领域和工业数据科学中面临的挑战。项目成员与科学合作者合作解决特定问题，必要时为Awkward Array添加功能，Anaconda和Nvidia的行业合作者更新Python的标准工具，以识别CPU和GPU工作负载的这些新阵列类型。这也是一个教育项目，向实践科学家和学生教授使用数组习惯用法解决复杂问题的新方法。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

Nowadays, scientists need to be programmers as well as experts in their fields. Any software that simplifies the computational gymnastics of analyzing data is welcomed, as it allows scientists to focus more of their attention on what they want to compute, rather than how, but there is usually a tension between ease of use and computational speed. Faster computation means more data analyzed, but error-free code matters most. Awkward Array is a software library that performs calculations on data that do not fit into neat rows. For instance, a fleet of undersea probes may each measure ocean temperatures at a different number of depths, but data analysis tools like Excel, Pandas, and SQL require measurements to be arranged in rectangular tables with the same number of columns in each row. The problem is more acute when variable numbers of measurements are nested within variable numbers of entities. Traditionally, scientists have either used slow scripting languages or unforgiving "bare metal" languages to perform calculations on these non-tabular datasets. Awkward Array generalizes array concepts so that the easy and fast expressions that once applied only to rectangular tables now apply to irregularly shaped data, allowing bare metal speed in a high-level scripting language. This project broadens the applicability of Awkward Array beyond the problem domain it was originally intended for, particle physics, to a wide variety of scientific fields, including oceanography, astronomy, genetics, chemistry, and health care. It also integrates the Awkward Array library with popular data science and machine learning tools and extends the implementation to GPUs for extremely fast processing of the same array idioms.Scientists use NumPy-based tools, such as Pandas, to analyze large and regularly shaped data efficiently. Packing tables of numbers into contiguous arrays allows operations to be precompiled and fast; and expressing operations as concise commands with implicit loops makes them easier to read and quicker to type during data exploration. However, scientific data often has a complexity that does not fit well into tabular format. This forces scientists to write programs with explicit loops, which are slow in Python. But what about analysis of large and irregularly shaped data? Awkward Array is a generalization of NumPy; it is a Python library defining arrays of objects with arbitrary types and a suite of generic operations on them. Like NumPy arrays, Awkward Arrays are packed into contiguous buffers for efficiency, but unlike NumPy arrays, they can include variable-length lists, nested fields, missing values, and mixed types. Awkward Arrays use 10 times less memory than equivalent Python objects and are up to 100 times faster in computations. This project generalizes Awkward Array as a foundational library for science. It is a cross-disciplinary effort, extending Awkward Array to efficiently solve challenges faced by scientists in a variety of fields and data science in industry. The project members work with scientific collaborators to solve specific problems, adding features to Awkward Array if necessary, and industry collaborators at Anaconda and Nvidia update Python’s standard tools to recognize these new array types for CPU and GPU workloads. It is also an educational project, teaching new approaches to complex problems using array idioms, both to practicing scientists and to students.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.