EAGER: Truly Distributed Deep Learning: Representation and Computation

在许多科学领域，从医疗保健到天文学，我们收集数据的能力远远超过我们分析数据的能力。大多数数据分析算法要求所有数据都在一个中心位置可用，但由于数据的庞大规模或隐私问题（如医疗保健），这并不总是可能的。 该项目的目标是开发可以在分布式数据集上运行的数据分析算法，其中不同的物理位置包含数据的子集。 应用包括医疗诊断工具，因为它们基于比目前可能的大得多的数据集，因此更准确，以及众包数据分析任务，允许任何有空闲计算能力的人参与全球规模的计算。该项目有两个目标。 第一个是设计和实现一个本体支持的深度学习描述语言（DL 2），用于表示深度学习的所有阶段，包括模型结构、超参数和训练方法。DL 2将作为深度学习框架之间的中间语言，无论它们运行的硬件架构如何，都将支持模型共享，主要是为真正的分布式学习服务。DL 2的本体论基础将支持在共享模型时对框架兼容性的显式推理;对所有人开放的“模型动物园”，而不仅仅是特定框架的用户;以及对模型库进行语义查询的能力，例如，找到相似的模型。 第二个目标是设计、实现和彻底评估一些真正分布式的深度学习算法，这些算法利用DL 2进行模型共享。现有的分布式机器学习方法依赖于分布式算法，这些算法交换的浅层、紧凑模型比现代深度网络小几个数量级，这导致了分布式平均适应深度学习的有趣挑战。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响力评审标准进行评估，被认为值得支持。

In many scientific domains, from healthcare to astronomy, our ability to gather data far outstrips our ability to analyze it. Most data analysis algorithms require all of the data to be available at one central location, but that is not always possible due to either the sheer size of the data or, as in healthcare, privacy concerns. The goal of this project is to develop data analysis algorithms that can be run on distributed datasets, where different physical locations contain a subset of the data. Applications include medical diagnostic tools that are more accurate because they are based on significantly larger datasets than is currently possible, and crowdsourcing data analysis tasks by allowing anyone with some spare compute capacity to participate in a global-scale computation.The project has two aims. The first is the design and implement an ontologically backed Deep Learning Description Language (DL2) for representing all phases on deep learning, including model structure, hyperparameters, and training methods. DL2 will serve as an interlingua between deep learning frameworks, regardless of the hardware architecture on which they run, to support model sharing, primarily in service of truly distributed learning. The ontological underpinnings of DL2 will support, among other things, explicit reasoning about framework compatibility when sharing models; a "model zoo" that is open to all, not just users of a specific framework; and the ability to formulate semantic queries against model libraries to, for example, find similar models. The second aim is to design, implement, and thoroughly evaluate a number of truly distributed algorithms for deep learning that leverage DL2 for model sharing. Existing approaches to distributed machine learning rely on distributed algorithms that exchange shallow, compact models that are orders of magnitude smaller than modern deep networks, leading to interesting challenges in adapting distributed averaging to deep learning.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.