CSR: Small: Deconstructing Distributed Deep Learning

深度学习在计算机视觉、语音识别、自然语言处理等应用领域取得了长足的进步。新的算法、更大的数据集、更强的计算能力和机器学习框架都促成了这一成功。一个重要的缺失部分是，对于用户来说，有效地配置深度学习应用程序并将其集成到现有数据中心仍然是一项挑战。该项目开发了能够有效利用云资源的新解决方案，这反过来将有助于扩大用户群体，使其能够发现新的更好的深度学习模型，并将其应用于新的环境和应用。在开发新的应用程序时，用户尝试使用许多深度神经网络(DNN)，但对其计算需求了解有限。由于非线性扩展，预测吞吐量改善是具有挑战性的。本项目第一推力中开发的技术可快速指导资源调配和资源分配。在这样的环境中，有效的作业间资源共享(特别是对于类似的DNN)是一个悬而未决的问题，该项目的重点2通过开发有效的调度技术来解决。数据中心工作负载(DNN、Web)的多样性，以及不同的资源“亲和力”，创造了拥抱云联合的机会。这一项目致力于研究和教育的多样性，涉及本科生和研究生，再加上现有的广泛的K-12外联工作。研制的实验实验台集科研和教学于一体。所有的算法、设计、软件和数据都是公开的，以便研究人员和教育工作者能够复制和改进开发的技术。这些基本问题的解决是本项目的重点，有助于开发新的深度学习模型，并提高这些技术在新应用领域的采用率。所有报告和代码都存储在基于SVN的存储库中。软件和相关文档可在GitHub上公开获取。所有数据在项目生命周期后至少保存7年。研究产品在发布后立即可通过http://qed.usc.edu.获得，包括补充信息这些记录是持久的，可以通过标准的网络协议访问，并且是安全的。根据需要使用适当的存储介质，以保持数据访问的最新状态。支持该项目产生的专利的数据将在专利有效期内保留。资料库的网址是http://qed.usc.edu/D3/repository.This奖，反映了国家科学基金会的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Deep learning has made substantial strides in computer vision, speech recognition, natural language processing, and other applications. New algorithms, larger datasets, increased compute power, and machine learning frameworks all contribute to this success. An important missing piece is that it is still challenging for users to effectively provision and integrate deep learning applications into existing datacenters. This project develops novel solutions that enable effective use of cloud resources, which in turn will aid in broadening the population of users capable of discovering new and better deep learning models and applying them in novel settings and applications.When developing new applications, users experiment with many deep neural networks (DNNs), but have limited knowledge of their computational demand. Due to non-linear scaling, predicting throughput improvements is challenging. Techniques developed in thrust 1 of this project quickly guide provisioning and resource allocation. In such environments, efficient inter-job resource sharing (particularly for similar DNNs) is an open problem, addressed in thrust 2 of the project by developing effective scheduling techniques. The diversity of datacenter workloads (DNNs, web), with different resource "affinity", creates opportunities to embrace cloud federations. While promising, there is a lack of techniques to support their sustainable deployment; these are developed in thrust 3 of the project.This project is committed to diversity in research and education, involving undergraduate and graduate students, coupled with an existing extensive K-12 outreach effort. The developed experimental testbed is utilized for both, research and education. All algorithms, designs, software, and data are made publicly available so that researchers and educators are able to replicate and improve on developed technologies. Solutions to the fundamental problems that are the focus of this project enable the development of new deep learning models and increase the adoption rate of these technologies in novel application domains.All reports and code are stored in an SVN-based repository. Software and related documents are publicly available on GitHub. All data is kept for at least 7 years beyond the life of the project. Research products are available promptly after publication, including supplemental information, through http://qed.usc.edu. These records are durable, accessible through standard web protocols, and made secure. Appropriate storage media is used, to keep data access current, as needed. Data that supports patents resulting from the project is retained for the duration of the patents. The URL to the repository is http://qed.usc.edu/D3/repository.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.