Sparse Binary Transformers for Multivariate Time Series Modeling
Sparse Binary Transformers for Multivariate Time Series Modeling
复制标题
DOI：
10.1145/3580305.3599508
复制
发表时间：
2023-08
期刊：
影响因子：
--
通讯作者：
Matt Gorbett;H. Shirazi;I. Ray
中科院分区：
文献类型：
--
作者：
Matt Gorbett;H. Shirazi;I. Ray
压缩神经网络有潜力在新的应用程序和较小的计算环境中实现深度学习。然而，了解这些模型可以成功的学习任务的范围还没有得到很好的研究。在这项工作中，我们将稀疏和二进制加权变压器应用于多变量时间序列问题，表明轻量级模型的精度可与相同结构的密集浮点变压器相媲美。我们的模型在三个时间序列学习任务中取得了良好的结果：分类，异常检测和单步预测。此外，为了降低注意力机制的计算复杂度，我们应用了两个修改，这些修改几乎没有显示模型性能的下降：1）在分类任务中，我们将固定掩码应用于查询、键和值激活，以及2）用于预测和异常检测，其依赖于预测单个时间点的输出，我们提出了一个注意掩模，以允许仅在当前时间步进行计算。总之，每种压缩技术和注意力修改都大大减少了Transformer中所需的非零操作的数量。我们在一系列指标上测量了我们的方法的计算节省，包括参数计数，位大小和浮点运算（FLOPs）计数，显示存储大小减少了53倍，FLOPs减少了10.5倍。
Compressed Neural Networks have the potential to enable deep learning across new applications and smaller computational environments. However, understanding the range of learning tasks in which such models can succeed is not well studied. In this work, we apply sparse and binary-weighted Transformers to multivariate time series problems, showing that the lightweight models achieve accuracy comparable to that of dense floating-point Transformers of the same structure. Our model achieves favorable results across three time series learning tasks: classification, anomaly detection, and single-step forecasting. Additionally, to reduce the computational complexity of the attention mechanism, we apply two modifications, which show little to no decline in model performance: 1) in the classification task, we apply a fixed mask to the query, key, and value activations, and 2) for forecasting and anomaly detection, which rely on predicting outputs at a single point in time, we propose an attention mask to allow computation only at the current time step. Together, each compression technique and attention modification substantially reduces the number of non-zero operations necessary in the Transformer. We measure the computational savings of our approach over a range of metrics including parameter count, bit size, and floating point operation (FLOPs) count, showing up to a 53× reduction in storage size and up to 10.5× reduction in FLOPs.