Towards Improved Understanding and Efficient Utilization of Depthwise Computation in Modern Neural Networks

提高对现代神经网络深度计算的理解和有效利用

• 批准号: 577088-2022
• 负责人: Wolf, GuyG
• 金额: $ 3.28万
• 依托单位: Université de Montréal
• 依托单位国家: 加拿大
• 项目类别: Alliance Grants
• 财政年份: 2022
• 资助国家: 加拿大
• 起止时间: 2022-01-01 至 2023-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=743915
• 关键词: Towards; Improved; Understanding; Efficient; Utilization

Deep Neural Networks have shown remarkable performance in a wide array of tasks including image recognition, speech recognition, and text generation. They consist of a number of sequential modules typically jointly optimized for a specific task. The functional behavior of the individual modules that emerges in such systems is complex and difficult to understand or analyze, while at the same time yielding remarkable results. On the other hand, the computational and memory efficiency of deep networks is a large bottleneck to their widespread usability. In this project we propose to tackle these problems in a holistic manner. In the first step, we propose to develop and enhance analytical tools for comparing and analyzing intermediate representations of deep networks. Borrowing methods from kernel non-parametrics, our aim is to provide analytical tools to determine how representations evolve in depth, and what information is extracted at each layer. Ultimately we would like to use these tools to characterize an implicit objective function that emerges in depth. These aspects will be studied in both traditional feed-forward networks and modern networks with skip connections, as well as Transformer-based models utilizing self-attention layers. In the second step we will use these techniques to enhance local learning methods as part of principled local objective functions, targeted to allow increasing parallelism and saving memory when training large deep neural networks. Concurrently, we will investigate continuous-depth deep neural networks. Our work will utilize a variety of modern deep neural networks with a focus on the emerging Transformer architecture. Finally, we will apply the developed techniques in a broad set of application areas including natural language processing, computer vision, drug discovery, and recommender systems. Development of this will require interdisciplinary collective expertise in theoretical tools, model-parallel learning, and continuous depth networks. Over this three-year project the PI and co-investigators will train three PhD students and one MSc student across a wide range of industrially relevant skills, including deep learning and distributed learning.

深度神经网络在包括图像识别、语音识别和文本生成在内的各种任务中表现出了卓越的性能。它们由许多顺序模块组成，通常针对特定任务进行联合优化。在这样的系统中出现的各个模块的功能行为是复杂的，难以理解或分析，同时产生显着的结果。另一方面，深度网络的计算和内存效率是其广泛可用性的一大瓶颈。在这个项目中，我们建议以整体的方式解决这些问题。在第一步中，我们建议开发和增强分析工具，用于比较和分析深度网络的中间表示。借用内核非参数化的方法，我们的目标是提供分析工具，以确定如何表示深入发展，以及在每一层提取的信息。最终，我们想使用这些工具来表征一个隐含的目标函数，深入出现。这些方面将在传统的前馈网络和具有跳跃连接的现代网络以及利用自注意层的基于Transformer的模型中进行研究。在第二步中，我们将使用这些技术来增强局部学习方法，作为原则性局部目标函数的一部分，目标是在训练大型深度神经网络时增加并行性并节省内存。同时，我们将研究连续深度的深度神经网络。我们的工作将利用各种现代深度神经网络，重点关注新兴的Transformer架构。最后，我们将在广泛的应用领域中应用所开发的技术，包括自然语言处理，计算机视觉，药物发现和推荐系统。这将需要在理论工具，模型并行学习和连续深度网络的跨学科集体专业知识的发展。在这个为期三年的项目中，PI和合作研究人员将培训三名博士生和一名硕士生，涵盖广泛的工业相关技能，包括深度学习和分布式学习。