Collaborative Research: IIS: III: MEDIUM: Learning Protein-ish: Foundational Insight on Protein Language Models for Better Understanding, Democratized Access, and Discovery

大型语言模型是大规模的神经网络，它学习丰富的单词上下文表示，并使用这些表示来解决自然语言处理（NLP）中的各种任务。这些模型是生成式人工智能的一个突出例子，并且正在成为提取和组织大量生物数据库内容以及预测广泛的分子生物特性的有前途的方法。然而，我们对这些模型在其学习表征中捕获的内容知之甚少，为什么它们在某些任务上表现良好而在其他任务上表现不佳，以及它们如何对描述生物空间的关系产生深刻的见解。如果NLP的进展是任何迹象，那么当前通过大幅增加可训练参数数量来提高语言模型性能的趋势对于我们的碳足迹和确保学术环境中研究和奖学金的公平性/可及性来说都是不可持续的。该项目推进了信息集成和信息学交叉领域的算法研究，使用有原则的蛋白质语言模型（PLMs）作为计算工具，在不同的细节和规模水平上深入了解蛋白质空间的结构、功能和进化组织。它还旨在以一种资源意识、可持续和所有研究人员都可以使用的方式进行研究。研究活动分为三个重点：(1)在plm中编码先前的生物学知识，以便在复合空间中进行联合和资源感知学习；(2)揭示基本属性并组织学习到的表示空间，以便将捕获的内容与感兴趣的属性联系起来；(3)使plm能够捕获不同的背景，以便更深入地探索蛋白质空间中的结构、功能和进化组织。这种跨学科的方法为机器学习、生物信息学和分子生物学领域做出了贡献，并在这些学科的界面上为培养各级代表性不足的学生提供了机会。研究人员决心在社区和学科之间架起桥梁，他们计划开展活动，建立和激励一个跨学科的社区，以进一步推进他们的研究。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Large language models are massive neural networks that learn rich contextual representations of words and use such representations to address a variety of tasks in natural language processing (NLP). These models are a prominent example of generative artificial intelligence and are emerging as promising approaches for distilling and organizing the content of massive biological databases and for predicting a wide range of molecular bio-properties. Yet, we know surprisingly little about what these models capture in their learned representations, why they perform well on some tasks and not on others, and how they can produce deep insight into the relationships describing the biological space. If progress in NLP is any indication, the current trend of improving the performance of language models by drastically increasing the number of their trainable parameters is unsustainable both for our carbon footprint and for ensuring equity/accessibility of research and scholarship in the academic setting. This project advances algorithmic research at the intersection of information integration and informatics using principled protein language models (PLMs) as computational vehicles for deeper insight into the structural, functional, and evolutionary organization across protein space at varying levels of detail and scale. It also aims to do so in a way that is resource-aware, sustainable, and accessible to all researchers. The research activities are organized in three thrusts: (1) encoding prior biological knowledge in PLMs for joint and resource-aware learning in composite spaces, (2) revealing fundamental properties and organizing the learned representation space to inform and connect what is captured with properties of interest, and (3) enabling PLMs to capture diverse contexts for deeper exploration of the structural, functional, and evolutionary organization across protein space. This interdisciplinary approach contributes to the fields of machine learning, bioinformatics, and molecular biology and provides opportunities at the interface of these disciplines for training under-represented students of all levels. The investigators are determined to bridge communities and disciplines, and they have planned activities to build and galvanize a trans-disciplinary community to further advance their research.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.