III: Small: A New Machine Learning Paradigm Towards Effective yet Efficient Foundation Graph Learning Models

受基础语言模型在ChatGPT等应用中取得成功的启发，人们可以想象一个预训练的基础图学习模型（FGLM）在科学研究、社会网络分析、异常检测、药物发现和电子商务等领域的广泛应用所带来的深远影响。尽管预训练的图神经网络取得了重大进展，但目前还没有一个FGLM可以在各种与图学习相关的任务上达到预期的性能。为了弥补这一差距，该项目的目标是设计和开发一种新的机器学习范式（技术、方法和模型），以实现有效而高效的fglm，这将有助于研究人员和从业者在广泛的现实世界应用中推进他们的工作，这些应用由流行的图结构数据驱动，从而有助于增强国家安全、公共卫生和福利。项目成果（例如开源代码、基准数据和模型）将被公开访问，并通过演示、出版物、媒体演示等广泛分发。该项目将把研究与教育结合起来，包括新课程开发、学生指导、专业培训和劳动力发展，以及针对妇女和代表性不足群体的K-12外展活动。通过开发一种新的机器学习范式，在第一次尝试中共同解决图学习中的多任务、跨图和跨领域挑战，该项目包括三个相互关联的研究组成部分，以实现有效而高效的fglm。首先，为了以一种有效且经济的方式实现fglm强大且一致的任务泛化能力，给定一个图，团队将设计和开发一个新的多任务自监督图学习框架，该框架采用新颖的多梯度下降优化算法结合自适应数据增强，以公平地学习每个任务。其次，由于现实世界的图总是不完整的，为了学习特定领域的全面知识，团队将开发一种新的多图协同训练框架，即具有关系知识蒸馏的变分期望最大化框架，以有效而高效的方式联合训练生成的图，同时应对不同图的不同节点属性的挑战。第三，为了进一步实现预训练fglm的跨领域知识转移，该团队将开发基于混合专家的元学习技术，以表征来自不同领域的图的潜在属性，并自适应地利用从现有领域学习的知识来推断罕见或不可见的领域。本项目开发的新的机器学习范式将加速快速发展的预训练图神经网络领域的发展，推动信息集成和信息学领域的发展，并帮助不同领域的研究人员和实践者在无处不在的图数据驱动的各种现实世界应用中推进他们的工作。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Inspired by the success of foundation language models in applications such as ChatGPT, one can envision the far-reaching impacts that can be brought by a pre-trained Foundation Graph Learning Model (FGLM) with broader applications in the areas such as scientific research, social network analysis, anomaly detection, drug discovery, and e-commerce. Despite the significant progress of pre-trained graph neural networks, there has not yet a FGLM that can achieve desired performance on various graph-learning-related tasks. To bridge this gap, the goal of this project is to design and develop a new machine-learning paradigm (techniques, methods, and models) towards effective yet efficient FGLMs, which will help researchers and practitioners advance their work in a wide range of real-world applications driven by the prevalent graph-structured data, thus helping to enhance national safety, public health, and welfare. The project outcomes (such as open-source code, benchmark data, and models) will be made publicly accessible and be broadly distributed through demos, publications, media presentations, and the like. This project will integrate research with education, including novel curriculum development, student mentoring, professional training and workforce development, and K-12 outreach activities aimed at women and underrepresented groups.By developing a new machine-learning paradigm to jointly solve the multi-task, cross-graph, and cross-domain challenges in graph learning at the first attempt, this project includes three interconnected research components towards effective yet efficient FGLMs. First, to realize the strong and consistent task-generalization ability for FGLMs in an effective yet affordable way, given a graph, the team will design and develop a new multi-task self-supervised graph learning framework with a novel multi-gradient descent optimization algorithm coupled with adaptive data augmentation to learn each task equitably well. Second, as real-world graphs are always incomplete, to learn comprehensive knowledge for a specific domain, the team will develop a new multi-graph co-training framework, specifically a variational expectation-maximization framework with relational knowledge distillation, to jointly train the generated graphs in an effective yet efficient manner while tackling the challenge of diversified node attributes of different graphs. Third, to further enable cross-domain knowledge transfer for pre-trained FGLMs, the team will develop mixture-of-expert based meta-learning techniques to characterize the latent properties of graphs from different domains and adaptively utilize the knowledge learned from existing domains to infer on a rarely seen or unseen domain. The new machine-learning paradigm developed in this project will accelerate the development in the rapidly evolving area of pre-trained graph neural networks, advance the field of information integration and informatics, and help researchers and practitioners in different domains to advance their work in a variety of real-world applications driven by the ubiquitous graph data.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.