SCH: Human-Centered Reinforcement Learning for Personalized Nutritional Coaching

慢性疾病，如2型糖尿病、高血压和肥胖症，给个人和整个社会带来越来越大的负担。健康指导已经成为促进自我管理的有效方法。然而，没有足够的专业教练来适应不断增长的慢性病患者。会话代理有可能克服这些障碍，使健康教练提供给更多样化的人群。一种有前途的数据驱动方法采用强化学习（RL），这是一种机器学习方法，可以从过去的交互中学习，并规定达到预定目标的动作序列。然而，基于RL的对话可以被认为是不直观的用户，并有一个新的方法来调整基于RL的会话代理人与人类的推理和期望的需要。此外，强化学习算法是不透明的，需要新的方法来生成对强化学习推理和动作的解释。为了解决这些差距，该项目开发了一种新的方法来提供基于强化学习会话代理的健康教练，同时解决设计以人为本的基于强化学习会话代理的更普遍的挑战。为了实现这些目标，该项目包括在2型糖尿病背景下进行健康指导的用户研究，其中人类健康教练将被要求通过短信为2型糖尿病患者提供指导。在本研究中收集的对话语料库为开发文本膳食描述的数据驱动计算表示以及开发使用RL生成适合营养指导的对话结构的聊天机器人提供了基础。此外，该项目使用所学的膳食表征，为个人提供关于其营养选择的反馈和对这种反馈的解释。最后，它将人类视角集成到RL策略中，以生成人类直观感知的对话结构。评估研究检查了与其他非RL型教练技术相比，RL型健康教练对个人实现营养目标的能力的影响。这项研究在几个方面对整个社会产生了重大影响。首先，对话界面可以降低不同社区参与健康和健康技术干预的准入门槛，并减少健康方面的“干预产生的不平等”。此外，将RL与人类推理结合起来并向用户解释其推理和选择的新技术可以增加其对更广泛的问题和领域的适用性。在更广泛的层面上，该研究和教育计划为进一步推动以人为本的数据科学、机器学习和人工智能教育方法迈出了重要的一步，这些方法可以对该领域的未来研究产生更广泛的影响。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Chronic diseases, such as type 2 diabetes, hypertension, and obesity, place an ever-increasing burden on individuals and society at large. Health coaching has emerged as an effective approach to promoting self management. However, there are not enough coaching professionals to accommodate the growing population of individuals with chronic diseases. Conversational agents have the potential to overcome these barriers and make health coaching available to a more diverse population. One promising data-driven approach employs reinforcement learning (RL), a machine-learning approach that learns from past interactions and prescribes sequences of actions for reaching a predetermined goal. However, RL-based dialogs can be perceived as unintuitive to users, and there is a need for new approaches to aligning RL-based conversational agents with human reasoning and expectations. In addition, RL algorithms are opaque and there is a need for new approaches to generating explanations for RL inferences and actions.To address these gaps, this project develops a new approach to providing health coaching with RL-based conversational agents, while at the same time addressing more general challenges of designing human-centered RL-based conversational agents. To achieve these goals, this project includes a user study of health coaching in the context of type 2 diabetes, in which human health coaches will be asked to provide guidance to individuals with type 2 diabetes via text messages. The corpus of dialogs collected during this study provides a foundation for developing data driven computational representation of textual meal descriptions and for the development of a chatbot that uses RL to produce conversational structures appropriate for nutritional coaching. Furthermore, this project uses learned representations of meals to provide individuals with feedback on their nutritional choices and explanations for this feedback. Finally, it integrates the human perspective into the RL policy to generate dialog structures that are perceived as intuitive by humans. The evaluation study examines the impact of the RL-based health coach on individuals’ ability to achieve their nutritional goals as compared to other, non-RL-based coaching techniques. This research is consequential to society at large in several ways. First, conversational interfaces can lower entry barriers for engaging with technological interventions in health and wellness for diverse communities and reduce “intervention-generated inequalities” in health. Furthermore, new techniques for aligning RL with human reasoning and explaining its inferences and choices to users can increase its applicability to a broader set of problems and domains. On a broader level, this research and educational plan take important steps towards further promoting human-centered approaches to data science, machine learning, and artificial intelligence education that can have broader impact on future research in this field.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.