III: Small: A New Machine Learning Paradigm Towards Effective yet Efficient Foundation Graph Learning Models

受基础语言模型在ChatGPT等应用程序中的成功启发，可以预见预先训练的基础图学习模型(FGLM)可以带来深远的影响，并在科学研究、社交网络分析、异常检测、药物发现和电子商务等领域得到更广泛的应用。尽管预先训练的图神经网络已经取得了很大的进展，但还没有一种FGLM能够在各种与图学习相关的任务中达到预期的性能。为了弥补这一差距，本项目的目标是设计和开发一种新的机器学习范式(技术、方法和模型)，以实现有效而高效的FGLMS，这将帮助研究人员和实践者在流行的图形结构数据驱动的广泛现实世界应用中推进他们的工作，从而帮助提高国家安全、公共卫生和福利。项目成果(如开放源代码、基准数据和模型)将向公众公布，并通过演示、出版物、媒体演示等广泛分发。这个项目将把研究与教育结合起来，包括新的课程开发、学生指导、专业培训和劳动力发展，以及针对妇女和代表性不足的群体的K-12推广活动。通过开发新的机器学习范式来共同解决图表学习中的多任务、跨图和跨领域的挑战，该项目包括三个相互关联的研究组成部分，以实现有效而高效的FGLMS。首先，为了实现FGLMS强大而一致的任务泛化能力，在给定一个图的情况下，团队将设计和开发一个新的多任务自监督图学习框架，该框架结合新颖的多梯度下降优化算法和自适应数据扩充，以更好地学习每个任务。其次，由于现实世界的图总是不完整的，为了学习特定领域的综合知识，团队将开发一个新的多图联合训练框架，特别是带有关系知识蒸馏的变分期望最大化框架，在应对不同图多样化节点属性的挑战的同时，以有效而高效的方式联合训练生成的图。第三，为了进一步实现预先训练的FGLM的跨域知识转移，该团队将开发基于混合专家的元学习技术来表征来自不同领域的图的潜在属性，并自适应地利用从现有领域学习的知识来推断罕见或未见的领域。该项目开发的新的机器学习范式将加快快速发展的预先训练的图形神经网络领域的发展，推动信息集成和信息学领域的发展，并帮助不同领域的研究人员和实践者在无处不在的图形数据驱动的各种现实世界应用程序中推进他们的工作。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Inspired by the success of foundation language models in applications such as ChatGPT, one can envision the far-reaching impacts that can be brought by a pre-trained Foundation Graph Learning Model (FGLM) with broader applications in the areas such as scientific research, social network analysis, anomaly detection, drug discovery, and e-commerce. Despite the significant progress of pre-trained graph neural networks, there has not yet a FGLM that can achieve desired performance on various graph-learning-related tasks. To bridge this gap, the goal of this project is to design and develop a new machine-learning paradigm (techniques, methods, and models) towards effective yet efficient FGLMs, which will help researchers and practitioners advance their work in a wide range of real-world applications driven by the prevalent graph-structured data, thus helping to enhance national safety, public health, and welfare. The project outcomes (such as open-source code, benchmark data, and models) will be made publicly accessible and be broadly distributed through demos, publications, media presentations, and the like. This project will integrate research with education, including novel curriculum development, student mentoring, professional training and workforce development, and K-12 outreach activities aimed at women and underrepresented groups.By developing a new machine-learning paradigm to jointly solve the multi-task, cross-graph, and cross-domain challenges in graph learning at the first attempt, this project includes three interconnected research components towards effective yet efficient FGLMs. First, to realize the strong and consistent task-generalization ability for FGLMs in an effective yet affordable way, given a graph, the team will design and develop a new multi-task self-supervised graph learning framework with a novel multi-gradient descent optimization algorithm coupled with adaptive data augmentation to learn each task equitably well. Second, as real-world graphs are always incomplete, to learn comprehensive knowledge for a specific domain, the team will develop a new multi-graph co-training framework, specifically a variational expectation-maximization framework with relational knowledge distillation, to jointly train the generated graphs in an effective yet efficient manner while tackling the challenge of diversified node attributes of different graphs. Third, to further enable cross-domain knowledge transfer for pre-trained FGLMs, the team will develop mixture-of-expert based meta-learning techniques to characterize the latent properties of graphs from different domains and adaptively utilize the knowledge learned from existing domains to infer on a rarely seen or unseen domain. The new machine-learning paradigm developed in this project will accelerate the development in the rapidly evolving area of pre-trained graph neural networks, advance the field of information integration and informatics, and help researchers and practitioners in different domains to advance their work in a variety of real-world applications driven by the ubiquitous graph data.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.