Towards Improved Understanding and Efficient Utilization of Depthwise Computation in Modern Neural Networks

深度神经网络在包括图像识别、语音识别和文本生成在内的一系列任务中表现出了卓越的性能。它们由多个通常针对特定任务联合优化的顺序模块组成。这种系统中出现的各个模块的功能行为很复杂，很难理解或分析，但同时也产生了显著的结果。另一方面，深层网络的计算和存储效率是其广泛使用的一大瓶颈。在这个项目中，我们建议以整体的方式解决这些问题。在第一步中，我们建议开发和增强用于比较和分析深层网络的中间表示的分析工具。借用核非参数的方法，我们的目标是提供分析工具来确定表示如何深入演变，以及在每一层提取什么信息。最终，我们想要使用这些工具来描述一个隐含的目标函数，该函数深入地出现。这些方面将在传统前馈网络和具有跳过连接的现代网络中进行研究，以及使用自我关注层的基于变压器的模型。在第二步中，我们将使用这些技术来增强局部学习方法，作为原则性局部目标函数的一部分，目标是在训练大型深度神经网络时允许增加并行性和节省内存。同时，我们还将研究连续深度的深度神经网络。我们的工作将利用各种现代深度神经网络，重点放在新兴的变压器架构上。最后，我们将把开发的技术应用于广泛的应用领域，包括自然语言处理、计算机视觉、药物发现和推荐系统。这将需要理论工具、模型并行学习和持续深度网络方面的跨学科集体专业知识。在这个为期三年的项目中，PI和合作调查员将培训三名博士生和一名MSC学生，掌握广泛的与工业相关的技能，包括深度学习和分布式学习。

Deep Neural Networks have shown remarkable performance in a wide array of tasks including image recognition, speech recognition, and text generation. They consist of a number of sequential modules typically jointly optimized for a specific task. The functional behavior of the individual modules that emerges in such systems is complex and difficult to understand or analyze, while at the same time yielding remarkable results. On the other hand, the computational and memory efficiency of deep networks is a large bottleneck to their widespread usability. In this project we propose to tackle these problems in a holistic manner. In the first step, we propose to develop and enhance analytical tools for comparing and analyzing intermediate representations of deep networks. Borrowing methods from kernel non-parametrics, our aim is to provide analytical tools to determine how representations evolve in depth, and what information is extracted at each layer. Ultimately we would like to use these tools to characterize an implicit objective function that emerges in depth. These aspects will be studied in both traditional feed-forward networks and modern networks with skip connections, as well as Transformer-based models utilizing self-attention layers. In the second step we will use these techniques to enhance local learning methods as part of principled local objective functions, targeted to allow increasing parallelism and saving memory when training large deep neural networks. Concurrently, we will investigate continuous-depth deep neural networks. Our work will utilize a variety of modern deep neural networks with a focus on the emerging Transformer architecture. Finally, we will apply the developed techniques in a broad set of application areas including natural language processing, computer vision, drug discovery, and recommender systems. Development of this will require interdisciplinary collective expertise in theoretical tools, model-parallel learning, and continuous depth networks. Over this three-year project the PI and co-investigators will train three PhD students and one MSc student across a wide range of industrially relevant skills, including deep learning and distributed learning.