III: Small: Go Beyond Short-term Dependency and Homogeneity: A General-Purpose Transformer Recipe for Multi-Domain Heterogeneous Sequential Data Analysis

序列数据在医疗保健、网络安全、社会科学以及在线搜索和推荐系统等领域无处不在。深度学习技术最近在序列数据分析任务中取得了巨大的成功，例如序列预测、文本理解以及时间序列分类和聚类。然而，从许多领域创建的现代顺序数据正变得越来越庞大、复杂和特定于领域。在处理这种复杂的多领域序列数据时，现有的深度学习方法在捕获长期依赖性和推广到多个领域方面受到限制。为了弥合这一差距，该项目将开发原则性的算法和方法，可以处理数据异构性和长期依赖性，以分析复杂和多域的顺序数据。所提出的框架将是通用的各种类型的顺序数据，包括人类语言，时间序列和轨迹数据。它将为深度学习技术在社交网络分析、临床护理、智能交通、文本挖掘和自然语言处理等更具挑战性的序列数据分析应用中提供新的可能性。 具体而言，该项目旨在开发（i）基于变压器的多域点过程分析的无监督学习技术，（ii）基于变压器的度量学习技术，使大规模和多域时间序列分析成为可能，以及（iii）在预训练变压器上进行鲁棒和高效域迁移学习的技术。所开发的技术将通过解决这些问题的计算和统计挑战，同时享受捕获长期依赖性和数据异构性的计算效率和建模灵活性。该研究还将以易于使用的库的形式提供开源软件，这有助于相关领域的研究人员和从业人员分析复杂的序列数据。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Sequential data are ubiquitous in domains including healthcare, cyber security, social science, and online search and recommendation systems. Deep learning techniques have recently demonstrated tremendous successes in sequential data analysis tasks, such as sequential prediction, and text understanding, and times series classification and clustering. However, modern sequential data created from many domains are becoming ever more massive, complex, and domain-specific. When handling such complex and multi-domain sequential data, existing deep learning methods are limited in capturing long-term dependency and generalizing to multiple domains. To bridge such a gap, this project will develop principled algorithms and methodologies that can handle data heterogeneity and long-term dependencies for analyzing complex and multi-domain sequential data. The proposed framework will be generic to various types of sequential data including human language, time series, and trajectory data. It will open new possibilities of enabling deep learning techniques for more challenging sequential data analysis applications in social network analysis, clinical care, smart transportation, text mining, and natural language processing.The proposed framework leverages the popular transformer architecture and incorporates multi-domain adaptation to handle the data heterogeneity. Specifically, this project aims to develop (i) unsupervised learning techniques for transformer-based multi-domain point process analysis, (ii) transformer-based metric learning techniques that enable large-scale and multi-domain time series analysis, and (iii) techniques for robust and efficient domain transfer learning over pre-trained transformers. The developed techniques will enjoy both computational efficiency and modeling flexibility of capturing long-term dependency and data heterogeneity, by addressing the computational and statistical challenges for these problems. The proposed research will also deliver open-source software in the form of easy-to-use libraries, which facilitate researchers and practitioners in related fields to analyze complex sequential data.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.