面向新模型的深度神经网络求解器的共性组件关键技术研究：算法与性能提升

深度神经网络DNN给语音识别、视觉物体识别和文本处理等应用领域带来巨大成功。新型模型/算法不断涌现，其内核均需求解约束非凸优化问题，严重依赖于数据集规模、硬件平台和非线性优化的迭代求解性能。随着数据规模扩大和DNN模型参数增多，非线性优化求解器在GPU集群上的可扩展性和计算能力方面存在大的性能瓶颈。显著提升DNN求解器在大型异构硬件平台上的可扩展性和计算能力，关键在于子空间迭代，其严重依赖于优化的图剖分、Hessian矩阵预条件子。 计算资源共享的高计算性能，密切依赖于系统级网络调度和优化算法。本项目面向资源共享情形下的10^6核级异构硬件平台，针对历史/在线学习的海量数据，研究具有高可扩展性和高效计算能力的、10^12模型参数的DNN优化模型求解器，针对人脸图像数据集，进行并行软件实现和“数据输入流-DNN求解器-分布式平台”的一体化系统集成，形成大规模分布式深度学习求解软件。

The development of depth/reinforcement learning (DNN) brings great promise to practical applications such as speech recognition, visual object recognition and text processing. In recent years, utilising GPUs as a parallel method to solve deep learning problems is a significant achievement in recent years to tackle modestly sized deep networks in practice. The DNN parallel optimization algorithm design for GPUs, has great restrictions on sparsity of gradient operators (Hessian matrices) and amount and frequency of processes. It limits the scalability and adaptability of DNN parallel algorithm. Research on resource scheduling and network optimization algorithms for practical DNN solvers in case of large scale heterogeneous platform resource sharing has become the focus of algorithm theory research and application at home and abroad. This application is geared towards a large, heterogeneous, distributed computer hardware platform (10^4~10^6 processor core level), in the case of relatively abundant in computing and network resources and resource sharing, aiming at two types of data input streams: history learning and online learning, a parallel solver for DNN optimization model with high scalability and high computational power is studied. Complete the program of DNN optimization model parallel solver and integration software and hardware system by using face image data sets.