OAC Core: SMALL: DeepJIMU: Model-Parallelism Infrastructure for Large-scale Deep Learning by Gradient-Free Optimization

近年来，深度神经网络（DNN）的使用越来越多，以获得科学探索，商业管理，安全和医疗保健的有用见解。DNN模型性能的不断改进伴随着其复杂性和大小的增加，这表明了更大和更深的模型的明显趋势。这种趋势在许多重要的应用领域尤其如此，例如需要超高分辨率地理空间图像处理的遥感。这样的应用导致训练非常大的模型以适应单个计算设备（例如，图形处理单元GPU），并因此提出了在多个计算设备上划分这种模型和并行化训练过程的迫切需求（即，模型并行性）。然而，到目前为止，DNN的模型并行性还没有得到很好的探索，并且由于反向传播算法的固有瓶颈而非常困难，其中一层的训练密切依赖于所有先前层的输入。为了克服这些挑战，该项目旨在为基于不依赖反向传播进行训练的优化方法的大规模DNN提供一种全新的模型并行基础设施。该项目计划解决训练需要大量高维数据的非常大和非常深的神经网络模型的挑战。该项目将开发新的优化技术和分布式DNN训练软件基础设施，以实现模型并行深度学习训练的更广泛应用和部署。该项目包括教育和参与活动，将大大提高社区对分布式机器学习算法和系统的理解。这些活动包括教学和培训学生和同龄人，为研究生和本科生提供新课程，研究和实习机会，以及扩大代表性不足的群体和当地高中学生的参与。该项目汇集了机器学习算法，分布式计算系统，遥感和空间数据科学的研究人员，提高通过模型并行增强的深度学习应用程序的性能和可扩展性。具体来说，该项目的重点是提出和开发一套新的模型并行优化算法和系统基础设施，用于训练大规模DNN，特别是用于地理空间科学研究的海量数据集的图像处理。为了在训练中实现模型并行性，提出了新的无梯度优化方法，将DNN优化的整个问题分解为子问题，然后可以（由许多工人）以高效率并行单独解决。该项目的产品包括模型并行的新理论和算法，沿着高效的无梯度DNN训练框架，以及新的调度和工作平衡技术。具体而言，本项目的研究方向如下：1）开发新的无梯度方法来训练各种类型的DNN; 2）设计基于无梯度优化的模型并行化算法和理论框架;以及3）为广泛的模型并行DNN训练应用构建可扩展且高效的分布式训练框架，例如用于大型图的深度学习和用于图像处理的非常深的卷积神经网络。该项目还涉及新技术与当前最先进方法之间的理论和实验比较，包括使用基于梯度的优化和管道并行的方法。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

In recent years, the use of deep neural networks (DNNs) has been increasing to obtain useful insights for scientific explorations, business management, security, and healthcare. The constant improvement of DNN model performance has been accompanied by an increase in their complexity and size, which indicate a clear trend toward larger and deeper models. Such a trend is especially the case for numerous important application domains, such as remote sensing where super-high-resolution geospatial image processing is required. Such applications lead to a huge challenge for the training of very large models to fit on a single computing device (e.g., a graphics processing unit, GPU), and hence raises urgent demands for partitioning such models across multiple computing devices and parallelizing the training process (i.e., model parallelism). However, until now model parallelism for DNNs has been poorly explored and is very difficult due to the inherent bottleneck from the backpropagation algorithm, where the training of one layer closely depends on input from all the previous layers. To overcome these challenges, this project aims a radically new pathway toward model parallelism infrastructure for large-scale DNNs based on optimization methods that do not rely on backpropagation for training. This project plans to address the challenges of training very large and very deep neural network models that require huge amounts of high-dimensional data. The project will develop new optimization techniques and distributed DNN training software infrastructure to enable wider applications and deployment of model parallel deep learning training. The project includes educational and engagement activities that will greatly increase the community's understanding of distributed machine learning algorithms and systems. Those activities include teaching and training students and peers, providing graduate and undergraduate students with new courses, and research and internship opportunities, as well as broadening participation of underrepresented groups and students at local high schools.This project brings together researchers in machine learning algorithms, distributed computing systems, remote sensing, and spatial data science, to boost the performance and scalability of deep learning applications enhanced by model parallelism. Specifically, this project focuses on proposing and developing a suite of new model parallelism optimization algorithms and system infrastructure for training large-scale DNNs, especially for image processing of massive datasets for geospatial scientific research. To enable model parallelism in the training, new gradient-free optimization methods are proposed to break down the whole problem of DNN optimization into subproblems, which can then be solved separately in parallel (by many workers) with high efficiency. The products of this project include new theories and algorithms for model parallelism, along with an efficient gradient-free DNN training framework with new scheduling and work balancing techniques. Specifically, this project has the following research thrusts: 1) Develop new gradient-free methods for training various types of DNNs; 2) Designing an algorithmic and theoretical framework of model parallelization based on gradient-free optimization; and 3) Building a scalable and efficient distributed training framework for a broad range of model parallel DNN training applications, such as deep learning for large graphs and very deep convolutional neural networks for image processing. This project also involves both theoretical and experimental comparison between the new techniques and current state-of-the-art methods, including those using gradient-based optimizations and pipeline parallelism.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.