III: Small: Go Beyond Short-term Dependency and Homogeneity: A General-Purpose Transformer Recipe for Multi-Domain Heterogeneous Sequential Data Analysis

III：小：超越短期依赖性和同质性：用于多域异构顺序数据分析的通用 Transformer 配方

• 批准号: 2008334
• 负责人: Tuo Zhao
• 金额: $ 50万
• 依托单位: Georgia Tech Research Corporation
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-10-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2008334&HistoricalAwards=false
• 关键词: III; Small; Go; Beyond; Short

Sequential data are ubiquitous in domains including healthcare, cyber security, social science, and online search and recommendation systems. Deep learning techniques have recently demonstrated tremendous successes in sequential data analysis tasks, such as sequential prediction, and text understanding, and times series classification and clustering. However, modern sequential data created from many domains are becoming ever more massive, complex, and domain-specific. When handling such complex and multi-domain sequential data, existing deep learning methods are limited in capturing long-term dependency and generalizing to multiple domains. To bridge such a gap, this project will develop principled algorithms and methodologies that can handle data heterogeneity and long-term dependencies for analyzing complex and multi-domain sequential data. The proposed framework will be generic to various types of sequential data including human language, time series, and trajectory data. It will open new possibilities of enabling deep learning techniques for more challenging sequential data analysis applications in social network analysis, clinical care, smart transportation, text mining, and natural language processing.The proposed framework leverages the popular transformer architecture and incorporates multi-domain adaptation to handle the data heterogeneity. Specifically, this project aims to develop (i) unsupervised learning techniques for transformer-based multi-domain point process analysis, (ii) transformer-based metric learning techniques that enable large-scale and multi-domain time series analysis, and (iii) techniques for robust and efficient domain transfer learning over pre-trained transformers. The developed techniques will enjoy both computational efficiency and modeling flexibility of capturing long-term dependency and data heterogeneity, by addressing the computational and statistical challenges for these problems. The proposed research will also deliver open-source software in the form of easy-to-use libraries, which facilitate researchers and practitioners in related fields to analyze complex sequential data.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

序列数据在医疗保健、网络安全、社会科学以及在线搜索和推荐系统等领域无处不在。最近，深度学习技术在序列数据分析任务中取得了巨大的成功，例如序列预测、文本理解以及时间序列分类和聚类。然而，从许多领域创建的现代顺序数据正变得越来越海量、复杂和特定于领域。在处理这种复杂的、多领域的时序数据时，现有的深度学习方法在捕捉长期依赖关系和推广到多领域方面存在局限性。为了弥补这一差距，该项目将开发能够处理数据异构性和长期依赖关系的原则性算法和方法，以分析复杂和多领域的顺序数据。所提出的框架将适用于各种类型的序列数据，包括人类语言、时间序列和轨迹数据。它将为社交网络分析、临床护理、智能交通、文本挖掘和自然语言处理中更具挑战性的序列数据分析应用提供深度学习技术的新可能性。具体地说，该项目旨在开发(I)用于基于变压器的多域点过程分析的无监督学习技术，(Ii)基于变压器的度量学习技术，以实现大规模和多域时间序列分析，以及(Iii)用于对预先训练的变压器进行稳健和高效的域转移学习的技术。通过解决这些问题的计算和统计挑战，所开发的技术将享受捕捉长期相关性和数据异构性的计算效率和建模灵活性。这项拟议的研究还将以易于使用的库的形式提供开源软件，以方便相关领域的研究人员和从业者分析复杂的序列数据。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Differentiable Top-k Operator with Optimal Transport

• 发表时间: 2020-02
• 期刊: ArXiv
• 作者: Yujia Xie;H. Dai;Minshuo Chen;Bo Dai;T. Zhao;H. Zha;Wei Wei-Wei;Tomas Pfister

DyGen: Learning from Noisy Labels via Dynamics-Enhanced Generative Modeling

• DOI: 10.1145/3580305.3599318
• 发表时间: 2023-05
• 期刊: Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining
• 作者: Yuchen Zhuang;Yue Yu;Lingkai Kong;Xiang Chen;Chao Zhang

Why Do Deep Residual Networks Generalize Better than Deep Feedforward Networks? - A Neural Tangent Kernel Perspective

• 发表时间: 2020-02
• 期刊: ArXiv
• 作者: Kaixuan Huang;Yuqing Wang-;Molei Tao;T. Zhao

Transfer Learning of Graph Neural Networks with Ego-graph Information Maximization

• 发表时间: 2020-09
• 作者: Qi Zhu;Yidan Xu;Haonan Wang-;Chao Zhang;Jiawei Han;Carl Yang

BERTifying Hidden Markov Models for Multi-Source Weakly Supervised Named Entity Recognition

BERT 化隐马尔可夫模型以实现多源弱监督命名实体识别

• 发表时间: 2021
• 期刊: Annual Meeting of the Association for Computational Linguistics
• 作者: Li, Yinghao Li;Shetty, Pranav;Liu, Lucas;Zhang, Chao;Song, Le.
• 通讯作者: Song, Le.