EAGER: Knowledge-guided neurosymbolic AI with guardrails for safe virtual health assistants

该项目解决了生成性人工智能(AI)系统的局限性，特别是在用于支持医疗保健(VHA)的虚拟助手的背景下。虽然VHA显示出增强患者能力和解决临床专业知识短缺的潜力，但由于其输出不准确以及没有遵守相关的护理标准，对安全性和可获得性的担忧出现。为了缓解这些担忧，该项目提出了一种创新的方法，将临床协议和实践指南整合到人工智能系统中。这种方法将使安全受限的VHA得以发展，以支持临床医生并确保与患者的安全互动。此外，这种方法有助于提供临床医生友好的解释，促进人类和人工智能在医疗保健领域的合作。通过解决当前围绕生成性人工智能安全的重大担忧，这项研究将促进用户信心，并在需要人类-人工智能协作的安全关键领域采用。这项研究的成功可能会产生超越医疗保健的影响，例如融入交通规则的自动驾驶汽车或确保安全运营和维护合规的制造流程。此外，该项目与促进计算、劳动力发展和教育包容性的努力相一致。例如，为来自代表性不足背景的在校学生举办一年一度的人工智能夏令营，以及通过与跨学科人工智能职业相关的实习和劳动力发展模块与高中生、本科生和研究生接触。这项研究的主要创新在于利用知识图谱(KG)来构建护栏，帮助确保人工智能系统的安全。在与临床专家的合作下，将使用丰富的说明性知识(例如，医学术语和定义)以及程序性或过程知识(例如，诊断标准和临床实践指南)来指导神经处理体系结构，从而导致VHA的开发固有地被限制为安全的。此外，建议的相同方法还将使VHA能够生成终端用户(例如，临床医生)友好的解释，使系统可验证。该项目的两个重要成果将是：(A)有效地应用KG的医疗指南，以维持临床环境中的高安全标准，以及(B)使用与最终用户验证和决策相关的术语、概念和指南，生成最终用户易于理解的解释。该项目提出的核心技术将推动神经符号人工智能的最先进水平，朝着促进人类和人工智能之间强大的(可验证的和安全受限的)合作。这些进展有可能转移到具有安全关键应用的其他领域，从而为更广泛的人工智能研究领域及其更广泛的采用做出贡献。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This project addresses the limitations of generative artificial intelligence (AI) systems, particularly in the context of virtual assistants used to support healthcare (VHAs). While VHAs show potential for empowering patients and addressing clinical expertise shortages, concerns about safety and accessibility arise due to inaccuracies in their outputs and their lack of adherence to the relevant standards of care. To mitigate these concerns, the project proposes an innovative approach for integrating clinical protocols and practice guidelines within AI systems. This approach will enable the development of safety constrained VHAs that support clinicians and ensure safe interactions with patients. Additionally, the approach facilitates the provision of clinician-friendly explanations, fostering improved collaboration between humans and AI in healthcare. By addressing significant current concerns surrounding the safety of generative AI, the research will promote user confidence and adoption in safety-critical domains requiring human-AI collaboration. The research's success can have implications beyond healthcare, such as autonomous vehicles incorporating traffic rules or manufacturing processes ensuring safe operations and maintenance compliance. Furthermore, the project aligns with efforts to promote inclusivity in computing, workforce development, and education. Example initiatives include annual AI summer camp for school students from underrepresented backgrounds, and engagement with high school, undergraduate and graduate students through internships and workforce development modules relevant to interdisciplinary AI careers.The main innovation of this research lies in leveraging Knowledge Graphs (KGs) to construct guardrails that help ensure the safety of AI systems. In collaboration with clinical experts, a KG enriched with both declarative (e.g., medical terminology and definitions) and procedural or process knowledge (e.g., diagnostic criteria and clinical practice guidelines) will be employed to guide neural processing architectures, resulting in the development of VHAs inherently constrained to be safe. Furthermore, the same proposed methods will also equip VHAs with the ability to generate end user (e.g., clinician) friendly explanations making the system verifiable. The project's two important outcomes will be to (a) effectively apply medical guidelines from the KG to uphold high safety standards in a clinical setting, and (b) generate explanations that are easily comprehensible to end users using the terms, concepts, and guidelines relevant to end-user verification and decision-making. The core techniques proposed in this project will advance the state-of-the-art in neurosymbolic AI toward facilitating robust (verifiable and safety-constrained) collaboration between humans and AI. These advances have the potential for transferability to other domains with safety-critical applications, thus contributing to the broader field of AI research and its wider adoption.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.