TARNet: Task-Aware Reconstruction for Time-Series Transformer
TARNet: Task-Aware Reconstruction for Time-Series Transformer
复制标题
DOI：
10.1145/3534678.3539329
复制
发表时间：
2022-08
期刊：
影响因子：
--
通讯作者：
Ranak Roy Chowdhury;Xiyuan Zhang;Jingbo Shang;Rajesh K. Gupta;Dezhi Hong
中科院分区：
文献类型：
--
作者：
Ranak Roy Chowdhury;Xiyuan Zhang;Jingbo Shang;Rajesh K. Gupta;Dezhi Hong
时间序列数据包含时间顺序信息，其可以指导用于预测性结束任务的表示学习（例如，分类，回归）。最近，有一些尝试利用这样的顺序信息，首先通过重建随机掩蔽时间段的时间序列值来预训练时间序列模型，然后在同一数据集上进行最终任务微调，从而提高最终任务性能。然而，这种学习范式从最终任务开始就阻碍了数据重构。我们认为，以这种方式学习的表示不通知的最终任务，因此，可能是次优的最终任务的性能。事实上，不同的时间戳的重要性在不同的最终任务中可能会有很大的不同。我们认为，通过重建重要的时间戳来学习表征将是提高最终任务绩效的更好策略。在这项工作中，我们提出了TARNet（任务感知重建网络），这是一个使用Transformer来学习任务感知数据重建的新模型，可增强最终任务性能。具体来说，我们设计了一个数据驱动的掩蔽策略，该策略使用来自最终任务训练的自我注意力分数分布来采样最终任务认为重要的时间戳。然后，我们屏蔽这些时间戳的数据并重建它们，从而使重建任务感知。该重建任务在每个时期与最终任务交替训练，在单个模型中共享参数，允许通过重建学习的表示来提高最终任务的性能。在数十个分类和回归数据集上进行的广泛实验表明，在所有评估指标上，TARNet的性能都显着优于最先进的基线模型。
Time-series data contains temporal order information that can guide representation learning for predictive end tasks (e.g., classification, regression). Recently, there are some attempts to leverage such order information to first pre-train time-series models by reconstructing time-series values of randomly masked time segments, followed by an end-task fine-tuning on the same dataset, demonstrating improved end-task performance. However, this learning paradigm decouples data reconstruction from the end task. We argue that the representations learnt in this way are not informed by the end task and may, therefore, be sub-optimal for the end-task performance. In fact, the importance of different timestamps can vary significantly in different end tasks. We believe that representations learnt by reconstructing important timestamps would be a better strategy for improving end-task performance. In this work, we propose TARNet, Task-Aware Reconstruction Network, a new model using Transformers to learn task-aware data reconstruction that augments end-task performance. Specifically, we design a data-driven masking strategy that uses self-attention score distribution from end-task training to sample timestamps deemed important by the end task. Then, we mask out data at those timestamps and reconstruct them, thereby making the reconstruction task-aware. This reconstruction task is trained alternately with the end task at every epoch, sharing parameters in a single model, allowing the representation learnt through reconstruction to improve end-task performance. Extensive experiments on tens of classification and regression datasets show that TARNet significantly outperforms state-of-the-art baseline models across all evaluation metrics.