EAGER: Knowledge-guided neurosymbolic AI with guardrails for safe virtual health assistants

该项目解决了生成人工智能（AI）系统的局限性，特别是在用于支持医疗保健（VHA）的虚拟助理的背景下。虽然VHA显示出增强患者能力和解决临床专业知识短缺的潜力，但由于其输出不准确以及缺乏对相关护理标准的遵守，人们对安全性和可及性表示担忧。为了缓解这些担忧，该项目提出了一种创新的方法，将临床协议和实践指南整合到人工智能系统中。这种方法将能够开发安全受限的VHA，以支持临床医生并确保与患者的安全互动。此外，该方法有助于提供对临床医生友好的解释，促进人类与人工智能在医疗保健领域的合作。通过解决当前围绕生成式人工智能安全性的重大问题，该研究将促进用户在需要人类-人工智能协作的安全关键领域的信心和采用。该研究的成功可能会产生超出医疗保健范围的影响，例如自动驾驶汽车纳入交通规则或确保安全操作和维护合规性的制造过程。此外，该项目与促进计算，劳动力发展和教育包容性的努力保持一致。例如，为来自代表性不足背景的学生举办的年度人工智能夏令营，以及通过实习和与跨学科人工智能职业相关的劳动力发展模块与高中生、本科生和研究生进行互动。这项研究的主要创新在于利用知识图（KGs）构建护栏，帮助确保人工智能系统的安全。在与临床专家的合作下，一个既有陈述性（例如，医学术语和定义）和程序或过程知识（例如，诊断标准和临床实践指南）将用于指导神经处理架构，导致VHA的开发本质上受到安全性的限制。此外，所提出的相同方法还将使VHA具有生成最终用户（例如，临床医生）友好的解释，使系统可验证。该项目的两个重要成果将是：（a）有效应用幼儿园的医疗指南，以在临床环境中坚持高安全标准，以及（B）使用与最终用户验证和决策相关的术语、概念和指南，生成最终用户易于理解的解释。该项目提出的核心技术将推动神经符号AI的发展，促进人类和AI之间的强大（可验证和安全约束）协作。这些进步有可能转移到其他领域的安全关键应用，从而有助于更广泛的人工智能研究领域及其更广泛的采用。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估来支持。

This project addresses the limitations of generative artificial intelligence (AI) systems, particularly in the context of virtual assistants used to support healthcare (VHAs). While VHAs show potential for empowering patients and addressing clinical expertise shortages, concerns about safety and accessibility arise due to inaccuracies in their outputs and their lack of adherence to the relevant standards of care. To mitigate these concerns, the project proposes an innovative approach for integrating clinical protocols and practice guidelines within AI systems. This approach will enable the development of safety constrained VHAs that support clinicians and ensure safe interactions with patients. Additionally, the approach facilitates the provision of clinician-friendly explanations, fostering improved collaboration between humans and AI in healthcare. By addressing significant current concerns surrounding the safety of generative AI, the research will promote user confidence and adoption in safety-critical domains requiring human-AI collaboration. The research's success can have implications beyond healthcare, such as autonomous vehicles incorporating traffic rules or manufacturing processes ensuring safe operations and maintenance compliance. Furthermore, the project aligns with efforts to promote inclusivity in computing, workforce development, and education. Example initiatives include annual AI summer camp for school students from underrepresented backgrounds, and engagement with high school, undergraduate and graduate students through internships and workforce development modules relevant to interdisciplinary AI careers.The main innovation of this research lies in leveraging Knowledge Graphs (KGs) to construct guardrails that help ensure the safety of AI systems. In collaboration with clinical experts, a KG enriched with both declarative (e.g., medical terminology and definitions) and procedural or process knowledge (e.g., diagnostic criteria and clinical practice guidelines) will be employed to guide neural processing architectures, resulting in the development of VHAs inherently constrained to be safe. Furthermore, the same proposed methods will also equip VHAs with the ability to generate end user (e.g., clinician) friendly explanations making the system verifiable. The project's two important outcomes will be to (a) effectively apply medical guidelines from the KG to uphold high safety standards in a clinical setting, and (b) generate explanations that are easily comprehensible to end users using the terms, concepts, and guidelines relevant to end-user verification and decision-making. The core techniques proposed in this project will advance the state-of-the-art in neurosymbolic AI toward facilitating robust (verifiable and safety-constrained) collaboration between humans and AI. These advances have the potential for transferability to other domains with safety-critical applications, thus contributing to the broader field of AI research and its wider adoption.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.