CRII: RI: Self-Attention through the Bayesian Lens

CRII：RI：贝叶斯视角下的自注意力

• 批准号: 1850358
• 负责人: Liping Liu
• 金额: $ 17.5万
• 依托单位: Tufts University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-03-15 至 2021-02-28
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1850358&HistoricalAwards=false
• 关键词: CRII; RI; Self; Attention; through

Self-attention, a recently introduced augmentation to neural network architectures, has greatly improved neural network performance in a range of applications, particularly natural language processing and computer vision. Based conceptually on the way the human brain processes complex visual information by learning to selectively focus on the most salient elements, self-attention improves network ability to capture long-range relations in data. However, there is limited study in quantifying uncertainties of the outputs of self-attention networks, though uncertainty quantification is critically important for reliable learning models. The uncertainty of a self-attention network highly depends on where the network pays its attention to. This project will model the uncertainties associated with attention placement, and thereby better quantify the uncertainty in network outputs. The research will also convert part of the architecture design to standard computational procedures by utilizing statistical methods, and thus facilitate the design of new network architectures. This project has a secondary aim of applying the self-attention mechanism to statistical inference for computational efficiency. Ultimately, this project will produce new network architectures that are more reliable and more broadly applicable. This research will also support the development of a deep learning course for both graduate and undergraduate students at Tufts University. This project examines self-attention networks using a Bayesian approach and proposes a new modification - Bayesian Self-Attention Networks (BSANs). While self-attention networks use "attention weights" to take information from a special range of the data, BSANs assign probabilities to attention weights. By modeling uncertainties in the attention, BSANs naturally inherit desirable properties of Bayesian methods, such as better estimations of uncertainties and less overfitting of data. BSANs will automate the computation of attention probabilities as statistical inference procedures, simplifying the design of new attention-based neural networks, which will only need to determine where to place the attention structure. The study of BSANs will result in new network architectures, with the potential to improve reliability over a wide span of tasks in both natural language processing and graph data analysis. In addition to BSANs, this project will also use the self-attention mechanism to construct probability distributions that involve large numbers of variables. Being flexible and computationally efficient, the constructed distributions will be suitable for distribution approximation in large-scale statistical inference. This project will produce computationally efficient inference methods for Gaussian processes, a widely used model in machine learning and other related areas.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

自注意力是最近引入的神经网络架构增强功能，极大地提高了神经网络在一系列应用中的性能，特别是自然语言处理和计算机视觉。从概念上讲，基于人脑通过学习有选择地关注最显着元素来处理复杂视觉信息的方式，自注意力提高了网络捕获数据中的远程关系的能力。然而，尽管不确定性量化对于可靠的学习模型至关重要，但量化自注意力网络输出的不确定性的研究还很有限。自注意力网络的不确定性很大程度上取决于网络关注的位置。该项目将对与注意力放置相关的不确定性进行建模，从而更好地量化网络输出的不确定性。该研究还将利用统计方法将部分架构设计转换为标准计算程序，从而促进新网络架构的设计。该项目的第二个目标是将自注意力机制应用于统计推断，以提高计算效率。最终，该项目将产生更可靠、更广泛适用的新网络架构。这项研究还将支持塔夫茨大学研究生和本科生深度学习课程的开发。该项目使用贝叶斯方法检查自注意力网络，并提出了一种新的修改 - 贝叶斯自注意力网络（BSAN）。自注意力网络使用“注意力权重”从特定范围的数据中获取信息，而 BSAN 将概率分配给注意力权重。通过对注意力中的不确定性进行建模，BSAN 自然地继承了贝叶斯方法的理想特性，例如更好地估计不确定性和减少数据的过度拟合。 BSAN 将自动计算注意力概率作为统计推理过程，简化新的基于注意力的神经网络的设计，该网络只需要确定注意力结构的放置位置。 BSAN 的研究将产生新的网络架构，有可能提高自然语言处理和图形数据分析等广泛任务的可靠性。除了BSAN之外，该项目还将使用自注意力机制来构建涉及大量变量的概率分布。所构造的分布灵活且计算效率高，适用于大规模统计推断中的分布近似。该项目将为高斯过程提供计算高效的推理方法，高斯过程是机器学习和其他相关领域广泛使用的模型。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Localizing and Amortizing: Efficient Inference for Gaussian Processes

本地化和摊销：高斯过程的有效推理

• 发表时间: 2020
• 期刊: Proceedings of The 12th Asian Conference on Machine Learning
• 作者: Liu, Linfeng;Liu, Li-Ping
• 通讯作者: Liu, Li-Ping

Kriging Convolutional Networks

• DOI: 10.1609/aaai.v34i04.5716
• 发表时间: 2020-04
• 期刊: ArXiv
• 作者: G. Appleby;Linfeng Liu;Liping Liu