STrXL: Approximating Permutation Invariance/Equivariance to Model Arbitrary Cardinality Sets
STrXL: Approximating Permutation Invariance/Equivariance to Model Arbitrary Cardinality Sets
复制标题
DOI：
10.32473/flairs.37.1.135568
复制
发表时间：
2024-05
期刊：
影响因子：
--
通讯作者：
Kendra Givens;David Ludwig;Joshua L. Phillips
中科院分区：
文献类型：
--
作者：
Kendra Givens;David Ludwig;Joshua L. Phillips
当前处理集合的深度学习技术仅限于固定基数，当集合很大时，会导致计算复杂度急剧增加。为了解决这个问题，我们采用了用于对自然语言处理的长期依赖性进行建模的技术，并将它们与排列等变架构 Set Transformer (STr) 相结合。结果是 Set Transformer XL (STrXL)，这是一种新颖的深度学习模型，能够在给定固定计算资源的情况下扩展到任意基数集。 STrXL的扩展能力在于其循环架构。 STrXL 不是一次处理整个集合，而是一次仅处理集合的一部分，并使用内存机制来提供过去的附加输入。 STrXL 特别适用于处理 DNA 序列的高通量测序 (HTS) 样本集，因为它们的样本集大小可达数十万。当负责对 HTS 草原土壤样本和 MNIST 数字进行分类时，结果表明 STrXL 表现出预期的内存大小与精度权衡，该权衡与下游任务的复杂性成比例，但与 STr 不同的是，STrXL 能够泛化到任意基数集。
Current deep-learning techniques for processing sets are limited to a fixed cardinality, causing a steep increase in computational complexity when the set is large. To address this, we have taken techniques used to model long-term dependencies from natural language processing and combined them with the permutation equivariant architecture, Set Transformer (STr). The result is Set Transformer XL (STrXL), a novel deep learning model capable of extending to sets of arbitrary cardinality given fixed computing resources. STrXL's extension capability lies in its recurrent architecture. Rather than processing the entire set at once, STrXL processes only a portion of the set at a time and uses a memory mechanism to provide additional input from the past. STrXL is particularly applicable to processing sets of high-throughput sequencing (HTS) samples of DNA sequences as their set sizes can range into hundreds of thousands. When tasked with classifying HTS prairie soil samples and MNIST digits, results show that STrXL exhibits an expected memory size-accuracy trade-off that scales proportionally with the complexity of downstream tasks, but, unlike STr, is capable of generalizing to sets of arbitrary cardinality.