PFI-TT: A tool to automatically generate and optimize programs to operate on complex big data

该创新-技术转化合作伙伴关系（PFI-TT）项目的更广泛影响/商业潜力，使张量代数算法（TACO）的未来商业化成为可能，该算法利用稀疏数据处理来揭示隐藏的数据属性，从而改善许多领域。随着世界变得更加数字化和高度仪表化，稀疏数据集被捕获并存储在云上的集中存储库中。这些庞大的数据集存在于每一个可以想象的领域，例如日常交通信息、社会互动、购买模式、人口基因组学和环境监测。然而，分析和使用这些数据需要在稀疏数据集上以高性能和可扩展的方式使用许多复杂的张量和矩阵代数算法。到目前为止，最先进的系统过于复杂或能力有限，无法让从业人员充分利用这些数据集的潜力。这个PFI转化研究和技术开发项目将使用编译器技术，使稀疏数据处理简单，学生，科学家，数据分析师，TACO是第一个稀疏张量代数编译器，可以采取任何复杂的张量代数表达式（其中每个张量可以来自任何现有的稀疏和密集张量格式）并生成高性能中央处理器（CPU）和图形处理单元（GPU）代码，可与当前最先进的手工优化库相媲美。使用TACO，稀疏张量代数可以像稠密张量代数和数组代码一样进行编译转换和代码生成。拟议的项目将使TACO商业化，从而更容易分析复杂的数据。TACO的功能和简单性可能会带来更多的分析和更好的算法，进一步扩大经济和社会影响。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的知识价值和更广泛的影响审查标准进行评估来支持。

The broader impact/commercial potential of this Partnerships for Innovation - Technology Translation (PFI-TT) project in enabling the future commercialization of a Tensor Algebra Compiler (TACO) that utilizes sparse data processing to expose hidden data properties that can lead to improvements in many fields. As the world becomes more digitized and heavily instrumented, sparse data sets are captured and stored in centralized repositories on the cloud. These enormous data sets are in every conceivable area, such as daily traffic information, social interactions, purchasing patterns, population genomics, and environmental monitoring. However, analyzing and using this data requires the use of many complex tensor and matrix algebra algorithms on sparse data sets in a high-performance and scalable manner. Thus far, the state-of-the-art systems are too complicated or have limited capabilities to let practitioners harness the full potential of these data sets. This PFI translational research and technology development project will use compiler technology to make sparse data processing simple and accessible to students, scientists, data analysts, and engineers.TACO is the first sparse tensor algebra compiler that can take any complex tensor algebra expression (where each tensor can be from any existing sparse and dense tensor format) and generate high-performance Central Processing Unit (CPU) and Graphics Processing Unit (GPU) code comparable to the current state-of-the-art, hand-optimized libraries. With TACO, sparse tensor algebra can be put on the same compiler transformation and code generation footing as the dense tensor algebra and array codes. The proposed project will commercialize TACO enabling easier analysis of complex data. The power and simplicity of TACO may lead to more analysis and better algorithms, further expanding the economic and societal impacts.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.