SCH: Human-Centered Reinforcement Learning for Personalized Nutritional Coaching

慢性疾病，如2型糖尿病、高血压和肥胖，给个人和整个社会带来越来越大的负担。健康教练已经成为一种促进自我管理的有效方法。然而，没有足够的专业教练来适应越来越多的慢性病患者。会话代理有潜力克服这些障碍，使健康指导能够提供给更多样化的人群。一种有前景的数据驱动方法采用强化学习（RL），这是一种机器学习方法，可以从过去的交互中学习，并规定达到预定目标的行动顺序。然而，基于强化学习的对话可能会被用户认为是不直观的，因此需要新的方法来将基于强化学习的对话代理与人类的推理和期望结合起来。此外，强化学习算法是不透明的，需要新的方法来为强化学习推理和动作生成解释。为了解决这些差距，该项目开发了一种新的方法，通过基于强化学习的会话代理提供健康指导，同时解决了设计以人为本的基于强化学习的会话代理的更一般的挑战。为了实现这些目标，该项目包括一项2型糖尿病背景下健康指导的用户研究，其中人类健康教练将被要求通过短信为2型糖尿病患者提供指导。在本研究中收集的对话语料库为开发文本膳食描述的数据驱动计算表示以及开发使用强化学习生成适合营养指导的对话结构的聊天机器人提供了基础。此外，该项目使用学习到的膳食表征为个人提供关于他们的营养选择的反馈和对这种反馈的解释。最后，它将人类视角集成到RL策略中，以生成被人类视为直观的对话结构。评估研究考察了与其他非基于训练的教练技术相比，基于训练的健康教练对个人实现营养目标的能力的影响。这项研究在几个方面对整个社会产生了重大影响。首先，对话界面可以降低不同社区参与卫生和保健技术干预的进入门槛，并减少卫生领域“干预造成的不平等”。此外，将强化学习与人类推理结合起来并向用户解释其推论和选择的新技术可以增加其对更广泛的问题和领域的适用性。在更广泛的层面上，这项研究和教育计划为进一步促进以人为中心的数据科学、机器学习和人工智能教育方法迈出了重要的一步，这些方法可以对该领域的未来研究产生更广泛的影响。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Chronic diseases, such as type 2 diabetes, hypertension, and obesity, place an ever-increasing burden on individuals and society at large. Health coaching has emerged as an effective approach to promoting self management. However, there are not enough coaching professionals to accommodate the growing population of individuals with chronic diseases. Conversational agents have the potential to overcome these barriers and make health coaching available to a more diverse population. One promising data-driven approach employs reinforcement learning (RL), a machine-learning approach that learns from past interactions and prescribes sequences of actions for reaching a predetermined goal. However, RL-based dialogs can be perceived as unintuitive to users, and there is a need for new approaches to aligning RL-based conversational agents with human reasoning and expectations. In addition, RL algorithms are opaque and there is a need for new approaches to generating explanations for RL inferences and actions.To address these gaps, this project develops a new approach to providing health coaching with RL-based conversational agents, while at the same time addressing more general challenges of designing human-centered RL-based conversational agents. To achieve these goals, this project includes a user study of health coaching in the context of type 2 diabetes, in which human health coaches will be asked to provide guidance to individuals with type 2 diabetes via text messages. The corpus of dialogs collected during this study provides a foundation for developing data driven computational representation of textual meal descriptions and for the development of a chatbot that uses RL to produce conversational structures appropriate for nutritional coaching. Furthermore, this project uses learned representations of meals to provide individuals with feedback on their nutritional choices and explanations for this feedback. Finally, it integrates the human perspective into the RL policy to generate dialog structures that are perceived as intuitive by humans. The evaluation study examines the impact of the RL-based health coach on individuals’ ability to achieve their nutritional goals as compared to other, non-RL-based coaching techniques. This research is consequential to society at large in several ways. First, conversational interfaces can lower entry barriers for engaging with technological interventions in health and wellness for diverse communities and reduce “intervention-generated inequalities” in health. Furthermore, new techniques for aligning RL with human reasoning and explaining its inferences and choices to users can increase its applicability to a broader set of problems and domains. On a broader level, this research and educational plan take important steps towards further promoting human-centered approaches to data science, machine learning, and artificial intelligence education that can have broader impact on future research in this field.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.