III: Small: A New Machine Learning Paradigm Towards Effective yet Efficient Foundation Graph Learning Models

受ChatGPT等应用程序中基础语言模型的成功启发，可以想象预先训练的基础图学习模型（FGLM）可以带来深远的影响，在科学研究，社交网络分析，异常检测，药物发现和电子商务等领域具有更广泛的应用。尽管预训练的图神经网络取得了重大进展，但还没有一个FGLM可以在各种图学习相关的任务上实现所需的性能。为了弥合这一差距，该项目的目标是设计和开发一种新的机器学习范式（技术，方法和模型），以实现有效而高效的FGLM，这将有助于研究人员和从业人员在由流行的图结构数据驱动的广泛的现实世界应用中推进他们的工作，从而有助于提高国家安全，公共卫生和福利。项目成果（如开源代码、基准数据和模型）将通过演示、出版物、媒体演示等方式向公众开放并广泛传播。该项目将研究与教育相结合，包括新颖的课程开发，学生辅导，专业培训和劳动力发展，以及针对女性和代表性不足的群体的K-12外展活动。通过开发新的机器学习范式，首次尝试共同解决图学习中的多任务，跨图和跨领域挑战，该项目包括三个相互关联的研究组成部分，以实现有效而高效的FGLMs。首先，为了以一种有效而经济的方式实现FGLM强大而一致的任务泛化能力，给定一个图，该团队将设计和开发一个新的多任务自监督图学习框架，该框架具有一种新的多梯度下降优化算法，再加上自适应数据增强，以公平地学习每个任务。其次，由于现实世界的图总是不完整的，为了学习特定领域的综合知识，团队将开发一个新的多图协同训练框架，特别是具有关系知识蒸馏的变分期望最大化框架，以有效而高效的方式联合训练生成的图，同时应对不同图的多样化节点属性的挑战。第三，为了进一步实现预先训练的FGLM的跨领域知识转移，该团队将开发基于专家混合的元学习技术，以表征来自不同领域的图的潜在属性，并自适应地利用从现有领域学到的知识来推断一个罕见或未知的领域。该项目开发的新机器学习范式将加速预训练图神经网络快速发展领域的发展，推进信息集成和信息学领域，并帮助不同领域的研究人员和从业人员在各种真实的-该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准。

Inspired by the success of foundation language models in applications such as ChatGPT, one can envision the far-reaching impacts that can be brought by a pre-trained Foundation Graph Learning Model (FGLM) with broader applications in the areas such as scientific research, social network analysis, anomaly detection, drug discovery, and e-commerce. Despite the significant progress of pre-trained graph neural networks, there has not yet a FGLM that can achieve desired performance on various graph-learning-related tasks. To bridge this gap, the goal of this project is to design and develop a new machine-learning paradigm (techniques, methods, and models) towards effective yet efficient FGLMs, which will help researchers and practitioners advance their work in a wide range of real-world applications driven by the prevalent graph-structured data, thus helping to enhance national safety, public health, and welfare. The project outcomes (such as open-source code, benchmark data, and models) will be made publicly accessible and be broadly distributed through demos, publications, media presentations, and the like. This project will integrate research with education, including novel curriculum development, student mentoring, professional training and workforce development, and K-12 outreach activities aimed at women and underrepresented groups.By developing a new machine-learning paradigm to jointly solve the multi-task, cross-graph, and cross-domain challenges in graph learning at the first attempt, this project includes three interconnected research components towards effective yet efficient FGLMs. First, to realize the strong and consistent task-generalization ability for FGLMs in an effective yet affordable way, given a graph, the team will design and develop a new multi-task self-supervised graph learning framework with a novel multi-gradient descent optimization algorithm coupled with adaptive data augmentation to learn each task equitably well. Second, as real-world graphs are always incomplete, to learn comprehensive knowledge for a specific domain, the team will develop a new multi-graph co-training framework, specifically a variational expectation-maximization framework with relational knowledge distillation, to jointly train the generated graphs in an effective yet efficient manner while tackling the challenge of diversified node attributes of different graphs. Third, to further enable cross-domain knowledge transfer for pre-trained FGLMs, the team will develop mixture-of-expert based meta-learning techniques to characterize the latent properties of graphs from different domains and adaptively utilize the knowledge learned from existing domains to infer on a rarely seen or unseen domain. The new machine-learning paradigm developed in this project will accelerate the development in the rapidly evolving area of pre-trained graph neural networks, advance the field of information integration and informatics, and help researchers and practitioners in different domains to advance their work in a variety of real-world applications driven by the ubiquitous graph data.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.