CRII: CNS: Exploring Data and Model Sparsity in Deep Learning Systems using Graphs

由于深度学习模型规模的增加，大量的计算和存储资源专门用于它们的训练和部署。对此，该方案将稀疏性作为一个关键原语，在不牺牲模型精度的前提下，降低了对计算和存储资源的要求。它试图使用图形作为数据模型，在系统级别上调查对数据和模型中稀疏性的基本理解。初步研究收集了一些系统级的需求，并表明盲目采用稀疏线性代数和图分析系统领域的最佳实践，这两个领域都与深度学习无关，导致许多新的系统级开销用于深度学习计算。该研究方案将为稀疏数据提供更好的表示以提高数据的局部性，并将为解决涉及稀疏数据和模型的计算中的工作量不平衡问题开发理论和系统。该方案将减少内存消耗，并将实现更快的计算。该提案的这两个目标也将使资源受限的设备成为深度学习计算生态系统的一部分。深度学习使用于不同社会目标的用例成为可能，包括在医学、国防、金融欺诈检测等领域。通过启用更多样化的数据集和模型的用例(两者都是稀疏的)，该提案寻求通过在减少内存使用的情况下实现更快的计算来对社会产生更好的影响，从而降低各种深度学习模型的培训和部署成本。更直接的影响将体现在教育活动中，包括将已开发的学习工具(S)纳入课堂教学、学生项目、面向代表性不足的少数群体和女学生的外联活动。由于行业对这一领域的浓厚兴趣，学生们将接触到行业标准工具，并接受端到端深度学习系统和数据科学的培训--这是一个关键的行业要求，也是行业合作伙伴在NSF工作室中确定的。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Due to the increase in the size of deep learning models, huge computation and memory resources are dedicated for their training, and deployment. To this regard, the proposal treats sparsity as a key primitive to reduce the requirement on the computational and memory resources without sacrificing the accuracy of the models. It seeks to investigate the fundamental understanding of the sparsity in both the data and model at the system-level using graph as a data-model. The preliminary study gathered a few system-level requirements and show that blindly adopting the best practices of sparse linear algebra and graph analytics systems fields, both of which are unrelated to deep learning, lead to many new system-level overheads for deep learning computation. The research proposal will lay down a better sparse data representation to improve the data locality, and will develop theory and systems to solve the workload imbalance problem in the computation involving sparse data and models. The proposal will result in reduction of memory consumption and will achieve faster computation. Both the goals of the proposal will also enable resource-constrained devices to be a part of the deep learning computation ecosystem.Deep learning has enabled use-cases that are being used for diverse societal goals including in the fields of medical science, defense, financial fraud detection, etc. By enabling such use-cases of more diverse datasets and models, both sparse, the proposal seeks to create better impact on the society by achieving faster computation at reduced memory usage, resulting in reduced cost for training and deployment of various deep learning models. The more immediate impact will be in the educational activities in the form of incorporating the developed learning tool(s) for classroom teaching, student projects, outreach to under-represented, minority and female students. Due to the strong interest from the industry in this field, students will be exposed to industry standard tools, as well get trained on end-to-end deep learning systems and data science both-- a key industry requirement that was also identified by the industry partners in an NSF Workshop.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.