EAGER:Using Network Analysis And Representational Geometry To Learn Structure-Function Relationship In Neural Networks

EAGER：使用网络分析和表征几何来学习神经网络中的结构-功能关系

• 批准号: 2039741
• 负责人: Eva Dyer
• 金额: $ 20万
• 依托单位: Georgia Tech Research Corporation
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-01-01 至 2023-12-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2039741&HistoricalAwards=false
• 关键词: EAGER; Using; Network; Analysis; Representational

Over the past few years, neural networks have revolutionized the fields of computer vision and natural language processing and are now becoming commonplace in many scientific domains. Despite their successes, understanding how to design or build a neural network solution remains challenging and often results in a game of guess and check. This process is incredibly inefficient, and in the end, does not provide any insights into why a model is either good or bad. Thus, new approaches are needed to characterize the relationship between structure (how the network is constructed) and function (how the network performs on a task) in neural networks, and use this information to design learning systems that are more efficient and stable. The overarching goal of this project is to develop tools to model the relationship between the structure and function of deep neural networks. This project will generate a rich toolkit for extracting low-dimensional features from neural networks and will produce new insights that can be used to drive progress in the future design of systems capable of modifying their own architecture to adapt to new data streams.Given the dimensionality of the problem, the discovery of compact (low-dimensional) representations and metrics that can adequately capture signatures of ``learning'' will be critical. When learning is unsuccessful, these metrics will be used to diagnose problems inherent to the network structure, such as its depth, width, and density of connections. The first part of the project will use tools in network science to discover how concepts such as network sparsity or path diversity between inputs and outputs affect the network's learning performance and efficiency (e.g., the number of examples required to learn a modular task, or whether the network can learn continually without catastrophic forgetting). The second part of the project will develop tools to study how the geometry of representations formed within networks can be used to predict learning outcomes.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

在过去的几年里，神经网络已经彻底改变了计算机视觉和自然语言处理领域，现在在许多科学领域变得司空见惯。尽管他们取得了成功，但理解如何设计或构建神经网络解决方案仍然具有挑战性，并且经常导致猜测和检查的游戏。这个过程是非常低效的，并且最终不能提供任何关于一个模型是好是坏的见解。因此，需要新的方法来表征神经网络中结构（网络如何构建）和功能（网络如何执行任务）之间的关系，并使用这些信息来设计更高效和稳定的学习系统。这个项目的首要目标是开发工具来模拟深度神经网络的结构和功能之间的关系。该项目将生成一个丰富的工具箱，用于从神经网络中提取低维特征，并将产生新的见解，可用于推动未来设计能够修改自身架构以适应新数据流的系统的进展。考虑到问题的维度，能够充分捕获“学习”特征的紧凑（低维度）表示和度量的发现将是至关重要的。当学习不成功时，这些指标将用于诊断网络结构固有的问题，例如其深度、宽度和连接密度。该项目的第一部分将使用网络科学中的工具来发现诸如网络稀疏性或输入和输出之间的路径多样性等概念如何影响网络的学习性能和效率（例如，学习模块化任务所需的示例数量，或者网络是否可以在没有灾难性遗忘的情况下持续学习）。该项目的第二部分将开发工具来研究如何使用网络中形成的表征几何来预测学习结果。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

NISPA: Neuro-Inspired Stability-Plasticity Adaptation for Continual Learning in Sparse Networks

• DOI: 10.48550/arxiv.2206.09117
• 发表时间: 2022-06
• 期刊: ArXiv
• 作者: Mustafa Burak Gurbuz;C. Dovrolis