EAGER: Knowledge-guided neurosymbolic AI with guardrails for safe virtual health assistants

该项目解决了生成式人工智能（AI）系统的局限性，特别是在用于支持医疗保健（VHAs）的虚拟助手的背景下。虽然vha显示出增强患者权能和解决临床专业知识短缺问题的潜力，但由于其产出不准确以及缺乏对相关护理标准的遵守，人们对安全性和可及性产生了担忧。为了减轻这些担忧，该项目提出了一种创新方法，将临床协议和实践指南整合到人工智能系统中。这种方法将有助于开发安全受限的vha，以支持临床医生并确保与患者的安全互动。此外，该方法有助于提供对临床医生友好的解释，促进人类和人工智能在医疗保健领域的更好合作。通过解决当前围绕生成式人工智能安全性的重大问题，该研究将促进用户在需要人类与人工智能协作的安全关键领域的信心和采用。这项研究的成功可能会对医疗保健以外的领域产生影响，比如纳入交通规则的自动驾驶汽车，或者确保安全操作和维护合规的制造流程。此外，该项目与促进计算机、劳动力发展和教育包容性的努力相一致。例如，针对来自代表性不足背景的学生的年度人工智能夏令营，以及通过与跨学科人工智能职业相关的实习和劳动力发展模块，与高中生、本科生和研究生进行接触。该研究的主要创新在于利用知识图（KGs）构建有助于确保人工智能系统安全的护栏。在与临床专家的合作下，将采用一个既丰富了陈述性（如医学术语和定义）又丰富了程序性或过程性知识（如诊断标准和临床实践指南）的KG来指导神经处理架构，从而使vha的开发本质上受到安全约束。此外，同样提出的方法还将使vha能够生成最终用户（例如临床医生）友好的解释，使系统可验证。该项目的两个重要成果将是：(a)有效地应用KG的医疗指南，以在临床环境中维护高安全标准，以及(b)使用与最终用户验证和决策相关的术语、概念和指南，生成最终用户易于理解的解释。本项目提出的核心技术将推动神经符号人工智能的最新发展，促进人类与人工智能之间的强大（可验证和安全约束）协作。这些进展有可能转移到其他具有安全关键应用的领域，从而有助于更广泛的人工智能研究领域及其更广泛的采用。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This project addresses the limitations of generative artificial intelligence (AI) systems, particularly in the context of virtual assistants used to support healthcare (VHAs). While VHAs show potential for empowering patients and addressing clinical expertise shortages, concerns about safety and accessibility arise due to inaccuracies in their outputs and their lack of adherence to the relevant standards of care. To mitigate these concerns, the project proposes an innovative approach for integrating clinical protocols and practice guidelines within AI systems. This approach will enable the development of safety constrained VHAs that support clinicians and ensure safe interactions with patients. Additionally, the approach facilitates the provision of clinician-friendly explanations, fostering improved collaboration between humans and AI in healthcare. By addressing significant current concerns surrounding the safety of generative AI, the research will promote user confidence and adoption in safety-critical domains requiring human-AI collaboration. The research's success can have implications beyond healthcare, such as autonomous vehicles incorporating traffic rules or manufacturing processes ensuring safe operations and maintenance compliance. Furthermore, the project aligns with efforts to promote inclusivity in computing, workforce development, and education. Example initiatives include annual AI summer camp for school students from underrepresented backgrounds, and engagement with high school, undergraduate and graduate students through internships and workforce development modules relevant to interdisciplinary AI careers.The main innovation of this research lies in leveraging Knowledge Graphs (KGs) to construct guardrails that help ensure the safety of AI systems. In collaboration with clinical experts, a KG enriched with both declarative (e.g., medical terminology and definitions) and procedural or process knowledge (e.g., diagnostic criteria and clinical practice guidelines) will be employed to guide neural processing architectures, resulting in the development of VHAs inherently constrained to be safe. Furthermore, the same proposed methods will also equip VHAs with the ability to generate end user (e.g., clinician) friendly explanations making the system verifiable. The project's two important outcomes will be to (a) effectively apply medical guidelines from the KG to uphold high safety standards in a clinical setting, and (b) generate explanations that are easily comprehensible to end users using the terms, concepts, and guidelines relevant to end-user verification and decision-making. The core techniques proposed in this project will advance the state-of-the-art in neurosymbolic AI toward facilitating robust (verifiable and safety-constrained) collaboration between humans and AI. These advances have the potential for transferability to other domains with safety-critical applications, thus contributing to the broader field of AI research and its wider adoption.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.