PFI-TT: A tool to automatically generate and optimize programs to operate on complex big data

这个创新伙伴关系-技术翻译（PFI-TT）项目的更广泛的影响/商业潜力是使张量代数编译器（TACO）的未来商业化，该编译器利用稀疏数据处理来暴露隐藏的数据属性，可以导致许多领域的改进。随着世界变得更加数字化和大量仪器化，稀疏的数据集被捕获并存储在云上的集中存储库中。这些庞大的数据集存在于每一个可以想象到的领域，比如日常交通信息、社会互动、购买模式、人口基因组学和环境监测。然而，分析和使用这些数据需要在稀疏数据集上以高性能和可扩展的方式使用许多复杂张量和矩阵代数算法。到目前为止，最先进的系统过于复杂或能力有限，无法让从业者充分利用这些数据集的潜力。这个PFI转化研究和技术开发项目将使用编译器技术，使稀疏数据处理变得简单，并便于学生、科学家、数据分析师和工程师使用。TACO是第一个稀疏张量代数编译器，它可以采用任何复杂的张量代数表达式（其中每个张量可以来自任何现有的稀疏和密集张量格式），并生成与当前最先进的手动优化库相当的高性能中央处理单元（CPU）和图形处理单元（GPU）代码。利用TACO，稀疏张量代数可以与密集张量代数和数组代码放在相同的编译转换和代码生成基础上。拟议的项目将使TACO商业化，使复杂数据的分析更容易。TACO的强大和简单可能会带来更多的分析和更好的算法，进一步扩大经济和社会影响。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The broader impact/commercial potential of this Partnerships for Innovation - Technology Translation (PFI-TT) project in enabling the future commercialization of a Tensor Algebra Compiler (TACO) that utilizes sparse data processing to expose hidden data properties that can lead to improvements in many fields. As the world becomes more digitized and heavily instrumented, sparse data sets are captured and stored in centralized repositories on the cloud. These enormous data sets are in every conceivable area, such as daily traffic information, social interactions, purchasing patterns, population genomics, and environmental monitoring. However, analyzing and using this data requires the use of many complex tensor and matrix algebra algorithms on sparse data sets in a high-performance and scalable manner. Thus far, the state-of-the-art systems are too complicated or have limited capabilities to let practitioners harness the full potential of these data sets. This PFI translational research and technology development project will use compiler technology to make sparse data processing simple and accessible to students, scientists, data analysts, and engineers.TACO is the first sparse tensor algebra compiler that can take any complex tensor algebra expression (where each tensor can be from any existing sparse and dense tensor format) and generate high-performance Central Processing Unit (CPU) and Graphics Processing Unit (GPU) code comparable to the current state-of-the-art, hand-optimized libraries. With TACO, sparse tensor algebra can be put on the same compiler transformation and code generation footing as the dense tensor algebra and array codes. The proposed project will commercialize TACO enabling easier analysis of complex data. The power and simplicity of TACO may lead to more analysis and better algorithms, further expanding the economic and societal impacts.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.