EAGER: Truly Distributed Deep Learning: Representation and Computation

在许多科学领域，从医疗保健到天文学，我们收集数据的能力远远超过了我们分析数据的能力。大多数数据分析算法要求所有数据在一个中央位置可用，但由于数据的巨大大小或(如医疗保健)隐私问题，这并不总是可行的。该项目的目标是开发可以在分布式数据集上运行的数据分析算法，其中不同的物理位置包含数据的子集。应用包括更准确的医疗诊断工具，因为它们基于比目前可能的数据集大得多的数据集，以及通过允许任何拥有一些空闲计算能力的人参与全球范围的计算来众包数据分析任务。该项目有两个目标。首先是设计和实现一种本体支持的深度学习描述语言(DL2)，用于表示深度学习的所有阶段，包括模型结构、超参数和训练方法。DL2将作为深度学习框架之间的中间语言，无论它们运行在什么硬件架构上，以支持模型共享，主要是为真正的分布式学习服务。DL2的本体论基础将支持在共享模型时关于框架兼容性的显式推理；向所有人开放的“模型动物园”，而不仅仅是特定框架的用户；以及针对模型库制定语义查询的能力，例如，找到类似的模型。第二个目标是设计、实现和彻底评估一些真正的分布式深度学习算法，这些算法利用DL2进行模型共享。现有的分布式机器学习方法依赖于分布式算法，这些算法交换浅层、紧凑的模型，这些模型比现代深层网络小几个数量级，这导致了将分布式平均适应深度学习的有趣挑战。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

In many scientific domains, from healthcare to astronomy, our ability to gather data far outstrips our ability to analyze it. Most data analysis algorithms require all of the data to be available at one central location, but that is not always possible due to either the sheer size of the data or, as in healthcare, privacy concerns. The goal of this project is to develop data analysis algorithms that can be run on distributed datasets, where different physical locations contain a subset of the data. Applications include medical diagnostic tools that are more accurate because they are based on significantly larger datasets than is currently possible, and crowdsourcing data analysis tasks by allowing anyone with some spare compute capacity to participate in a global-scale computation.The project has two aims. The first is the design and implement an ontologically backed Deep Learning Description Language (DL2) for representing all phases on deep learning, including model structure, hyperparameters, and training methods. DL2 will serve as an interlingua between deep learning frameworks, regardless of the hardware architecture on which they run, to support model sharing, primarily in service of truly distributed learning. The ontological underpinnings of DL2 will support, among other things, explicit reasoning about framework compatibility when sharing models; a "model zoo" that is open to all, not just users of a specific framework; and the ability to formulate semantic queries against model libraries to, for example, find similar models. The second aim is to design, implement, and thoroughly evaluate a number of truly distributed algorithms for deep learning that leverage DL2 for model sharing. Existing approaches to distributed machine learning rely on distributed algorithms that exchange shallow, compact models that are orders of magnitude smaller than modern deep networks, leading to interesting challenges in adapting distributed averaging to deep learning.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.