RI:Small:Dynamic Networks for Efficient, Adaptive, and Multimodal Vision

RI:Small：用于高效、自适应和多模态视觉的动态网络

• 批准号: 2303153
• 负责人: Nuno Vasconcelos
• 金额: $ 60万
• 依托单位: University of California-San Diego
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-07-01 至 2026-06-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2303153&HistoricalAwards=false
• 关键词: RI; Small; Dynamic; Networks; Efficient

The recent introduction of a class of artificial intelligence (AI) methods known as deep learning has enabled transformational advances in computer vision. It is now possible to imagine AI systems capable of understanding and responding to a visual scene in a manner similar to people. However, much research is still needed to enable effective solutions for perceptual tasks combining multiple modalities (vision, audio, language), with great robustness to domain variability, and high efficiency in terms of computation and energy consumption. While unified solutions to these problems are critically important for applications such as robotics or the deployment of AI on edge devices, such as cellphones, the problems are usually addressed independently, leading to disconnected solutions. Hence, there is a need for basic research in neural network architectures that not only advance each of the problems per se but also contribute to the integrated solution of all problems. Drawing inspiration from the plasticity of biological brains, the project will explore the use of dynamic networks, which vary their computations depending on their input, to achieve this goal. The challenge is that, because it is impossible to dynamically predict the massive number of parameters of a modern network, this requires careful selection of the parameters to endow with dynamics. The project aims to develop algorithms for such parameter selection and investigate their benefits for a range of applications in computer vision.To achieve its objectives, the project will leverage the well-known importance of feature mixing in deep learning, implementing network dynamics efficiently via feature-based attention mechanisms that perform mixing through dynamic matrix factorizations. These are factorizations of layer weight matrices that enable the restriction of dynamic parameters to a small latent kernel per layer. Two classes of factorization are proposed, enabling the implementation of dynamic attention by parameter synthesis or non-linear transformations. These factorizations are then proposed as a unified substrate for the design of perception algorithms with state-of-the-art performance for multimodal perceptual tasks, such as visual grounding or audio-visual spatial localization. These algorithms will be implementable with extremely light weight deep learning networks to enable high computation and energy efficiency, and are applicable across architectures that range from convolutional networks (CNNs) to transformers. Finally, they will have strong robustness to variability of operating environments and so can support challenging applications such as home robotics or perception on edge devices. The research has applicability in areas of societal relevance, such as manufacturing, self-driving vehicles, intelligent systems, assisted living, homeland security, etc. Educationally, the project will provide exciting opportunities for both graduate and undergraduate research.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

最近引入了一类称为深度学习的人工智能（AI）方法，使计算机视觉取得了变革性的进展。现在可以想象人工智能系统能够以类似于人类的方式理解和响应视觉场景。然而，仍然需要进行大量的研究，以实现有效的解决方案，结合多种形式（视觉，音频，语言）的感知任务，具有很大的鲁棒性域的变化，并在计算和能源消耗方面的高效率。虽然这些问题的统一解决方案对于机器人技术或在手机等边缘设备上部署人工智能等应用至关重要，但这些问题通常是独立解决的，从而导致解决方案脱节。因此，有必要对神经网络架构进行基础研究，不仅要推进每个问题本身，还要有助于所有问题的综合解决方案。从生物大脑的可塑性中汲取灵感，该项目将探索使用动态网络，根据其输入来改变其计算，以实现这一目标。挑战在于，由于不可能动态预测现代网络的大量参数，因此需要仔细选择参数以赋予动态性。该项目旨在开发用于此类参数选择的算法，并研究其在计算机视觉领域的一系列应用中的优势。为了实现其目标，该项目将利用深度学习中众所周知的特征混合的重要性，通过基于特征的注意力机制有效地实现网络动态，该机制通过动态矩阵分解执行混合。这些是层权重矩阵的因子分解，其使得能够将动态参数限制为每层的小潜在内核。提出了两类因子分解，使动态注意力的参数合成或非线性变换的实施。这些分解，然后提出了一个统一的基板与国家的最先进的性能多模态感知任务，如视觉接地或视听空间定位的感知算法的设计。这些算法将可通过极轻的深度学习网络实现，以实现高计算和能源效率，并适用于从卷积网络（CNN）到变压器的各种架构。最后，它们将对操作环境的变化具有很强的鲁棒性，因此可以支持具有挑战性的应用，例如家庭机器人或边缘设备上的感知。该研究在社会相关领域具有适用性，如制造业、自动驾驶汽车、智能系统、辅助生活、国土安全等。在教育方面，该项目将为研究生和本科生的研究提供令人兴奋的机会。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估来支持。